Fabian Delph ended his four-year wait for his first Barclays Premier League goal - and it couldn't have been better timed.
The former Leeds United midfield hit a stunning 80th-minute drive to give Villa a superb victory after they had twice been pegged back at St Mary's.
And he also set up the first effort for Gabby Agbonlahor.
It was a special night for Agbonlahor, too, the club's leading Premier League scorer hitting his first goal of the season to open the scoring in the 15th minute.
Villa led at the interval through Agbonalhor's well-taken effort but the home side quickly exposed the loss of skipper Ron Vlaar at half-time with a headed equaliser from Jay Rodriguez three minutes into the second period.
Kozak restored the advantage with a 64th-minute header, only for sub Dani Osvaldo to head a second equaliser five minutes later.
But Villa stormed back again, Delph firing home from 22 yards following a fine pass from Karim El Ahmadi.
There were a couple of scares in the opening minutes.
Vlaar blocked a close-range Adam Lallana shot following a mistake by Leandro Bacuna, and then Lallana's looping cross from the left dropped on top of the bar.
Brad Guzan had some difficulty in pushing out a tremendous 25-yard drive in the 14th minute, but a minute later the visitors were ahead.
Agbonlahor cleverly allowed Delph's pass from deep inside the Villa half to run past Nathaniel Clyne on the half-way line, allowing him a clear path to goal.
He cut neatly inside Dejan Lovren inside the penalty area and clipped a delicate shot over the advancing Paulo Gazzaniga.
In Villa's next attack, Nathan Baker was well off target with a header from Ashley Westwood's right-wing corner, but it needed a spectacular 27th-minute save from Guzan to keep Villa's lead intact, the keeper diving to his right to tip away a Rodriguez header.
As the Saints continued to press, a timely interception by Ciaran Clark denied Rodriguez another scoring opportunity before Guzan did well to hold Luke Shaw's powerful 30-yard effort.
Although Southampton had enjoyed greater possession throughout the opening period, there was frustration for Villa just before the break when Agbonlahor broke through the middle, only to penalised for handball.
The striker was adamant he had done nothing wrong, however, and was still arguing the point as the teams went off.
There was a surprise when Vlaar didn't appear for the second half, Matt Lowton taking over on the right of the three-man central defence, and in the 48th minute Southampton were level.
Although El Ahmadi was only just wide of the far post with a low angled shot which might have doubled Villa's lead, the home side equalised when Clyne crossed from the right and Rodriguez beat Guzan with a perfectly placed header which dropped just inside the right-hand post.
Guzan did well to react when sub Osvaldo produced a cheeky back-heel following a low cross from the left, but in the 64th minute Villa were back in front.
The tenacious El Ahmadi dispossessed Jack Cork on the right touchline and the midfielder's superb cross was met by Kozak, who sent a firm header towards goal. Gazzaniga got a hand to it but could only help the ball over the line.
The celebrations, though, lasted just five minutes, Rickie Lambert heading a deep right-wing cross back across goal for Osvaldo to head home from a couple of yards.
But after Guzan had saved well from another Rodriguez header, Lambert's battlers were in front for a third time with a classic counter attack in the 80th minute.
Having been under pressure, they broke quickly and El Ahmadi picked out Delph with a well-measured pass.
In the closing minutes Lambert sent a fierce free-kick into Villa's defensive wall and also had a short deflected for a corner, but the visitors held firm for a memorable victory.
Southampton: Gazzaniga; Clyne (Fonte 58), Yoshida, Lovren, Shaw; Davis, Wanyama (Cork HT), Ward-Prowse (Osvaldo 38); Lallana, Lambert, Rodriguez. Unused subs: Cropper, Chambers, Hooiveld, Gallagher.
Villa: Guzan; Vlaar (Lowton HT), Baker, Clark; Bacuna, El Ahmadi (Weimann 85) , Westwood, Delph, Luna; Agbonlahor, Kozak (Benteke 66. Unused subs: Steer, Albrighton, Sylla, Tonev.