He hopes to finish the swim in 50 days but the marathon challenge could be his toughest yet.
He hopes to complete his 350-mile swim in 50 days - wearing just a cap, trunks and goggles - in order to raise awareness of the need for ocean conservation.
His quest is part of the worldwide Action for Oceans campaign, which calls on governments to fully protect at least 30% of the world's oceans by 2030.
Pugh has spent months preparing in the sea off Cape Town, where he lives.
On the eve of the swim, he wrote: "This is the longest swim anyone has ever done, adhering to Channel Rules in water this low temperature.
"I honestly don't know how I'm going to do this. But I do believe I'll see the white cliffs of Dover.
The 48-year-old endurance athlete has swum the waters of the Arctic, the Antarctic and any number of other seas, rivers, streams and lakes.
He has even done a high-altitude paddle on Mount Everest but has previously said that this swim will be one of his biggest challenges.
"I've swum in some of the most dangerous and coldest waters on Earth but I rank this [swim] a 10 out of 10 for difficulty," he said.
From Land's End in Cornwall, Pugh will head east, battling the currents and frigid temperatures for up to five hours a day.
He hopes to make Dover in 50 days but the tides, headwinds, and sheer loneliness of open water swimming may have the last word.
Some 1,800 people have done the 21-mile (33km) crossing between England and France, but Pugh will attempt the entire 348 miles of water officially designated as the English Channel.