South Africa, set a revised target of 176 from 29 overs, were 104-5 in 21 overs when a heavy downpour at the Premadasa stadium forced the match to be called off.
With 121-5 being the par score at that stage, the Proteas lost by 17 runs to suffer their second defeat in the series following the 180-run loss in the first match last Saturday.
Left-arm spinner Herath trapped opener Alviro Petersen leg-before for 24, and had Faf du Plessis caught behind to reduce the Proteas to 69-5 by the 14th over.
David Miller (22 not out) and Ryan McLaren (14 not out) put on 35 runs for the unbroken sixth wicket before rain ruined their hopes of pulling off an upset.
South Africa were dealt another blow as star batsman Hashim Amla injured his left knee while fielding and was rushed to hospital for precautionary scans.
Amla, who had been sidelined for the first match due to a strained back, was unable to bat and faces a race against time to be fit for the third match in Pallekele on Friday.
Earlier, fast bowler Morne Morkel claimed 3-34 as the hosts were restricted to 223-9 in 49.2 overs after they won the toss and elected to bat.
Sri Lanka's stand-in captain Dinesh Chandimal and opener Tillakaratne Dilshan top-scored with 43 apiece before heavy rain ended the innings with four deliveries still to be bowled.
South Africa regrouped after the heavy defeat in the first match to keep the free-stroking Sri Lankan batsmen in check on the slow wicket.
Sri Lanka lost opener Upul Tharanga in the fourth over before a second wicket stand of 59 between Dilshan and Kumar Sangakkara helped them recover to 66-1 in the 18th over.
Sangakkara, who smashed a career-best 169 on Saturday, hit six boundaries in his 37 before he mistimed a drive off left-arm spinner Aaron Phangiso and was caught in the covers.
South Africa pegged the hosts down to 123-4 in 30 overs as Mahela Jayawardene was bowled by Robin Peterson for 17 and Dilshan was caught behind off Morkel soon after.
Chandimal, leading Sri Lanka in place of the banned Angelo Mathews, boosted the total with four boundaries in his 51-ball innings.
Two Sri Lankan umpires officiated after Australia's Rod Tucker, who was due to stand at one end, fell ill before the start.
