In the parliamentary poll the National Front lost its two-thirds majority - needed to make constitutional changes - and control of four state assemblies.
It did, however, win a simple majority, taking 139 out of 219 seats, with three more seats yet to declare.
Analysts blame ethnic tensions, crime and inflation for a drop in popularity.
Prime Minister Abdullah Ahmad Badawi is urging calm, amid fears there may be violence in the wake of the result.
Opposition figure Anwar Ibrahim hailed the result as a message that it was time for change in Malaysia.
This is a defining moment, unprecedented in our nation's history 
Mr Anwar's Justice Party has 31 seats out of the opposition's 82 so far, making him the leader of the opposition.
Mr Abdullah, in office since 2003, said he would meet the constitutional monarch on Monday and ask to form a new government. He dismissed suggestions that he would now face pressure from party members to step down.
His son-in law, Khairy Jamaluddin, told reporters: "We suffered a lot of losses tonight. But we are going to fight on. We are not going to quit. It is not the end of the world and we are going to get through this."
'Time for change'
The BBC's Robin Brant says no one expected the opposition to do so well across the board.
It is clear, he adds, that people wanted change and Chinese and Indian ethnic minority voters deserted the National Front, in power since 1957.
Before the elections only one state was under opposition control, Kelantan.
The Election Commission confirmed opposition wins in Kelantan as well as Selangor, Perak, Kedah and Penang.
Mr Anwar said the people of Malaysia had spoken.
"Today at the ballot box, you listened to your heart with a lot of conviction that the time for change has arrived..." he said.
"This is a defining moment, unprecedented in our nation's history. Today a new chapter has opened."
Ethnic minorities
Mr Anwar has accused the government of widespread vote-rigging.
Our correspondent says there are many people who have as many suspicions about Mr Anwar as about the National Front's leaders. But, he adds, the claim that Malaysia has free and fair elections is not a just one.
Ethnic minorities make up more than a third of the population. Many complain that government policy has denied them fair access to jobs, education, and housing.
Growing tensions between minority communities and the Malay majority have dominated the election campaign and the government has appealed for calm.
The last time the National Front suffered a big setback, in 1969, it resulted in race riots, dozens of deaths and a state of emergency.
Clashes
Some violence linked to the election was reported in the east of the country on Saturday.
Police in Terennganu State said they had fired tear gas to disperse a crowd of several hundred people protesting at what they saw as vote-rigging.
Supporters of the opposition PAS stopped buses they suspected of carrying National Front coalition supporters pretending to be voters from the district, said local police chief Ayob Yaacob.
He said that 22 people had been arrested and the rest of the crowd ran away.
RACHMAD YULIADI NASIR
Bank Muamalat Indonesia (BMI)Arthaloka : 913-22775-99
www.the-independent-news.blogspot.com
www.halamansatu.net/index.php?option=com_content&task=view&id=349&itemid=50
