Affleck held hands with Sam—who was barefoot and wrapped up in a towel—while leaving the local Pacific Palisades pool. The actor was casual