score:6
On the schedule you linked, there are two buttons that say "LEAVING / RETURNING". If you click the RETURNING button, you will see departure times from East 34th St. The first departure on weekends is at 6:30 AM.
There is also a route map showing clearly that it backtracks: from Wall Street to Brooklyn to Midtown, then back to Brooklyn and then Wall Street.