The first ten minutes welcomed some sustained pressure from City, which led to a free-kick just outside East Fife's 18-yard box.
Steven Mackay's curling strike was stopped by a full stretch John Dodds.
Then the influential Ryan Blackadder picked the ball up just over the halfway line in the 30th minute and threaded a delicate ball straight through the City defence, where Craig O'Reilly struck the ball past Kieron Renton to give the Fifers the lead.
Seven minutes later a disorganised City defence tried frantically to clear the ball but, with so many bodies in the box, the ball ricocheted to the dangerous Blackadder, who was taken down by a flying slide tackle from Elgin keeper Renton.
A penalty was awarded on the 37th minute but Renton made up for his mistake by saving an O'Reilly strike to deny East Fife a two-goal lead.
The second half saw three goals within six minutes, two for Elgin and one for East Fife.
Craig Smart headed into an empty Elgin net to put East Fife 2-0 in front.
Elgin quickly retaliated with an Adam Moffat's sliding challenge, sending the ball into the back of the net after a scramble in the East Fife box.
City's second goal came from a quick free-kick which dropped to a swivelling Paul Kaczan, who volleyed the ball to the left of the keeper.
East Fife once again took the lead in the 70th minute, an O'Reilly volley securing the victory.
Elgin City have suffered ten games without a win, but a performance like this against top of the table East Fife may be a positive sign for the future.