Deputy police chief in Ha Tinh province, Bui Dinh Quang, said Thursday that all the victims were Vietnamese subcontractors hired to work on a seaport breakwater project.
He said it was not known if any other workers were trapped in the rubble. Police and rescue workers are still searching through the rubble from the Wednesday night accident.
Online newspaper VnExpress quoted local official Pham Tran De as saying the scaffolding collapsed when the workers were making a giant concrete block for the breakwater.
The newspaper quoted survivor Hoang Huu Thai as saying there were more than 40 people working on the scaffolding at that time.
"The scaffolding was shaking one or two times before collapsing," Thai was quoted as saying. "Some workers were panicking, but somebody calmed us down and everyone continued working. About 20 minutes after the first shaking, the scaffolding suddenly collapsed. I thought I would die. I don't know who pulled me from the rubble."
The Son Duong seaport is part of the Vung Ang economic zone where Taiwan's Formosa Plastics Corp. is building a multi-billion dollar steel complex.
The site was the scene of an anti-China protest last May where killing three Chinese nationals were killed following China's deployment of an oil rig in disputed waters in the South China Sea.