A hotly-disputed 50th minute penalty cost Fleet this match and sent them sliding to their 5th consecutive league defeat and closer to the relegation zone.
Fleet have now lost 12 of their last 14 league matches.
The visit to 4th place Walton & Hersham, Holden's former club, was always going to be difficult, but once again defensive errors brought about another defeat.
Fleet started well and enjoyed plenty of possession early on. From one early move, striker Josh Francois went close, just shooting wide when clear through.
The half looked to be ending goal-less when from a free-kick, Walton's, Adam Thompson rose unchallenged, and Scott Edgar fired in Thompson's blocked shot from a few yards out.
Fleet then equalised immediately after the re-start. Bernard Asante sliding the ball home from a cross from the left from Tinoi Christie.
But just five minutes later Andy Darnton was adjudged to have brought down striker, Tristan Frontin and Jamie Revie made no mistake from the penalty spot.
