Despite having ssh enable on both ESXi 5.0 servers, I found that I could copy to the server from a desktop client or non-ESXi server without issue. However, when I wanted to copy between ESXi servers it did not work. It would just timeout:

ssh: connect to host la-host04 port 22: Connection timed out

I have discovered that the issue is the Security Profile on each of the ESXi servers. Through the vSphere client, you need to click on the ESXi server, then click on the Configuration tab. From here, select Security Profile, and click on the Firewall Properties …

Then, you just need to check off the SSH Client under Secure Shell. This will open up ssh communication between ESXi servers. You can restrict the settings by accessing the Firewall Settings for any of the services.

To enable from the command line using esxcli:
To list your current configuration:
# esxcli network firewall ruleset list

To enable sshClient to enable copying using ssh:
# esxcli network firewall ruleset set –ruleset-id sshClient –enabled yes

To refresh the configuration:
# esxcli network firewall refresh

For ESXi 5.1 Update 1:
# esxcli network firewall ruleset set -e true -r sshClient