Top seed Williams, who pulled out of the clay court tournament in the first round on her debut last year, got off to a positive
start before seeing the match go in Petrova's favour for a 4-6, 6-2, 6-3 victory.
Older sister Venus will be confirmed as the number two player in women's tennis behind Serena when the WTA rankings are next published Monday, after she reached the Madrid quarter-finals on Wednesday.
However Serena's failure to get past the third round here will give her rivals hope only 10 days before the start of the French Open at Roland Garros.
Having only narrowly beaten another Russian, Vera Dushevina, on Monday, Williams looked to be heading to certain victory after world number 18 Petrova had to have a long massage after losing the first set.
However the Russian re-emerged to take control of the next set and, as Williams failed to produce a challenge, the third set and the match.
Number 16 seed Petrova, who beat Williams in their last encounter, at Beijing in September 2009, will now meet Czech Lucie Safarova or Romanian Alexandra Dulgheru in the quarter-finals.
Just R20 for the first month. Support independent journalism by subscribing to our digital news package.