The 53-year-old Liu, a writer who was previously jailed over the 1989 Tiananmen pro-democracy protests, has been charged with "inciting subversion of state power" after co-authoring a bold call for political reform.
Western diplomats have been denied access to the hearing in a Beijing court after Washington and Brussels last week called for Liu´s release, and key dissidents said they were warned not to try to attend the proceedings.
If convicted, Liu -- who has already been detained for a year -- faces a maximum of 15 years in prison.
Dozens of police officers ringed the courthouse and sealed off the pavements, as a handful of diplomats and Liu supporters gathered outside.
The dissident´s wife Liu Xia told AFP by telephone that her brother, who attended the proceedings, said the trial had ended without a verdict.
That information was later confirmed to reporters by a court official, who did not give his name.
Liu Xia herself was prevented from attending the hearing by police stationed outside her home, she said.
She said, quoting her brother, that Liu had thanked her for her support before the court. She then burst into tears and hung up the phone.
"Long live democracy, long live Liu Xiaobo!" shouted 49-year-old Beijing resident Song Zaimin, under the watchful eye of police outside the court.
"I don´t know him and I don´t want to know him but I just support his heroic behaviour. We must stand up for our future."
Liu, a university professor before his involvement in the Tiananmen demonstrations, co-authored Charter 08, which calls for human rights protection and the reform of China´s one-party communist system.
The petition, which has been widely circulated online, has been signed by more than 10,000 people, including other key dissidents and intellectuals, according to China Human Rights Defenders, a network of activists.
It specifically calls for the abolition of subversion in China´s criminal code -- the very crime with which Liu has been charged.
"I have the right to express my opinions about society and politics," prominent human rights lawyer Teng Biao, a Charter 08 signatory, told reporters at the courthouse.
"If Liu Xiaobo is sentenced, then we should also assume the same crime, and we should bear the same legal responsibility."
Diplomats from more a dozen countries including the United States, Britain, Canada, Sweden, Australia and New Zealand stood outside the courthouse, awaiting news.
The United States and the European Union appealed for Liu´s unconditional release and an end to the harassment and detention of political dissidents -- calls which Beijing dismissed as "unacceptable".
US embassy official Gregory May reiterated Washington´s stance at the courthouse.
"We call on the government of China to release him immediately and to respect the rights of all Chinese citizens to peacefully express their political views," May said, noting the case had been raised "at high levels".
Liu´s trial comes amid fears that officials are attempting to rush the case through the court during the Western holiday season in a bid to attract less global attention.
The subversion charge -- which Liu´s lawyer Shang Baojun says also relates to articles posted on the Internet -- is routinely brought against those who voice opposition to China´s ruling Communist Party.
Rights groups accuse Beijing of abusing such charges to silence critics.
"The only purpose of this trial is to dress up naked political repression in the trappings of legal proceedings," said Sophie Richardson, Asia advocacy director at Human Rights Watch.
"Liu´s crimes are non-existent, yet his fate has been pre-determined. This is a travesty of justice."
Related story
When will the closed trial centers reopen?
