Damn You Windows Firewall!

One particular cause of the following error is finally known:

sockRecv(47ms):
    WindowsError: An existing connection was forcibly closed by the remote host.
    WindowsErrorCode: 0x2746
    numBytesRequested: 5
    Failed to receive data on the TCP socket
--sockRecv
sockRecv failed.
sockRecvN_buf: Did not receive the exact number of bytes desired.
numBytesToReceive: 5

This error was caused by the Windows Firewall doing stateful FTP filtering. The solution is to disable stateful FTP filtering so that the firewall does not block any FTP traffic. (This is typically prescribed for the server-side of FTP, but it also applies to the client-side.)

Note: The Windows Firewall’s stateful FTP filtering only causes problems when the connection is SSL/TLS. Clear (non-secured) connections are not affected.

To disable stateful FTP filtering, open an administrative command prompt, and type the following:

netsh advfirewall set global StatefulFTP disable

Also Note: The above error could also be a result of the connection truly being closed by the remote host, or something on the server-side. This solution only applies to the case where Windows Firewall (on the client-side computer) is interfering with a SSL/TLS FTP connection.

How to Reproduce the Problem

To reproduce the problem, write a program that establishes a TLS connection with an FTP server.   After authenticating, write a simple loop that calls ftp.Noop 1500 times.   If Windows Firewall is interfering, it is likely to fail in the same place every time.  For example, in my case the error occurs on iteration 699 every time.

Also, the error may not be easily reproducible even if Windows Firewall is a problem.  For example, the problem seemed to be not reproducible if the FTP server was not a Windows-hosted FTP server.   One might suspect the server-side to be the problem, but the connection reset ([RST, ACK] in a WireShark trace) definitely originated from the client-side.

Disabling the firewall’s Stateful Packet Inspection (SPI) feature solved the problem every time. (It should be the case that stateful packet inspection should NEVER try to inspect the packets of a TLS encrypted channel for the simple fact that it’s impossible to inspect.  The firewall does not have the ability to decrypt the packets in the channel, and therefore it cannot inspect anything. The firewall’s only choices SHOULD be to simply allow or disallow the connection.  I don’t understand why stateful packet inspection should ever be “monkeying around” with encrypted channels..)

v9.4.1.43 Fixed FTP2 XCRC Problem

Fixed an FTP2 XCRC issue. If the AutoXcrc property was turned on, Chilkat FTP2 was not computing the CRC of the local file (or data) uploaded to the server. This caused the CRC comparison to fail after the XCRC command was issued.

Note: This fix only applies when using AutoXcrc in conjunction with a server that actually supports XCRC.

This is a pre-release fix. Pre-release builds are made available upon request (support@chilkatsoft.com). Please make sure to specify the exact build needed (programming language, operating system, architecture, .NET Framework, Visual Studio Version, or anything else that may be required to know exactly which build is required).

FTP Timeouts on Large Files

Here is a description of the problem from the FileZilla documentation at http://wiki.filezilla-project.org/Network_Configuration#Timeouts_on_large_files

A few possible workarounds are described below.

Timeouts on large files

If you can transfer small files without any issues, but transfers of larger files end with a timeout, a broken router and/or firewall exists between the client and the server and is causing a problem.

As mentioned above, FTP uses two TCP connections: a control connection to submit commands and receive replies, and a data connection for actual file transfers. It is the nature of FTP that during a transfer the control connection stays completely idle.

The TCP specifications do not set a limit on the amount of time a connection can stay idle. Unless explicitly closed, a connection is assumed to remain alive indefinitely. However, many routers and firewalls automatically close idle connections after a certain period of time. Worse, they often don’t notify the user, but just silently drop the connection. For FTP, this means that during a long transfer the control connection can get dropped because it is detected as idle, but neither client nor server are notified. So when all data has been transferred, the server assumes the control connection is alive and it sends the transfer confirmation reply. Likewise, the client thinks the control connection is alive and it waits for the reply from the server. But since the control connection got dropped without notification, the reply never arrives and eventually the connection will timeout.

In an attempt to solve this problem, the TCP specifications include a way to send keep-alive packets on otherwise idle TCP connections, to tell all involved parties that the connection is still alive and needed. However, the TCP specifications also make it very clear that these keep-alive packets should not be sent more often than once every two hours. Therefore, with added tolerance for network latency, connections can stay idle for up to 2 hours and 4 minutes.

However, many routers and firewalls drop connections that have been idle for less than 2 hours and 4 minutes. This violates the TCP specifications (RFC 5382 makes this especially clear). In other words, all routers and firewalls that are dropping idle connections too early cannot be used for long FTP transfers. Unfortunately manufacturers of consumer-grade router and firewall vendors do not care about specifications … all they care about is getting your money (and only deliver barely working lowest quality junk).

To solve this problem, you need to uninstall affected firewalls and replace faulty routers with better-quality ones.

Some Possible Workarounds

1) Set the SoSndBuf and SoRcvBuf properties equal to 131072, which is the default used by FileZilla. If this improves the transfer rate, then it may be that the transfer takes a shorter amount of time and the timeout of whatever is interfering is not exceeded.

2) Set the SkipFinalReply property = true (or 1 if using the ActiveX). This will cause the FTP2 object to avoid trying to get the final control-channel reply. However, you should (A) close the connection and re-connect because if the final control-channel reply did arrive but remained unread, then the FTP client and server will be out-of-sync in terms of commands and replies. (In other words, the next FTP command sent on the control channel will get the previous reply.) (B) When using SkipFinalReply, write code to verify the size of the remote file after the transfer.

ListPattern Property Setting can Affect how FTP2 Gets Directory Information

If an FTP server supports the RFC 3659 extension, which includes the MLSD command, then it is automatically known to Chilkat FTP2 because the FTP server’s response to the FEAT command will include MLSD in the list. (Chilkat automatically sends a FEAT command after connecting so that it knows the capabilities of the FTP server.)

When Chilkat FTP2 recognizes that MLSD is supported, it will use this extension whenever possible. The reason is that MLSD returns more accurate date/time and size information. The drawback is that it can only return a full directory listing. Therefore, if the ListPattern property is set to anything other than “*”, such as “*.txt”, then MLSD cannot be used.

When MLSD is not used, Chilkat must use the FTP protocol’s LIST command which returns the directory listing in a human-readable format. The problem with the LIST response is that different operating systems and FTP servers respond with different listing formats. Chilkat FTP2 automatically recognizes and parses all known directory formats, including formats for uncommon systems such as VAX/VMS, MVS, AS/400, etc. Another issue with the human-readable response is that date/time information may be inaccurate (for example, timezone may not be known, or for older files the HH:MM:SS may not be provided).

Therefore, when using FTP2 to examine the contents of a remote directory, if accurate date/time information is important, then make sure the ListPattern is set to “*” and write your application code to iterate over all the files in the remote directory, perhaps only acting on the files that have names matching what your app needs.

Of course, even when using “*”, if the FTP server itself does not support MLSD, then your app is at the mercy of whatever information is or is not included in the human readable directory listing returned by the LIST command.

SFTP vs. FTPS

Clarification on the acronyms “SFTP” and “FTPS”

“SFTP” is the Secure File Transfer Protocol over SSH.  It is a protocol unrelated to the FTP protocol. (It is actually a subsystem of SSH.)  The Chilkat SSH / SFTP component is used for “SFTP”.  SFTP is achieved by connecting to an SSH server at port 22.

On the other hand, the Chilkat FTP2 component is for FTP.   FTP servers listen at port 21 (non-SSL/TLS) and port 990 (SSL).  FTP over SSL (i.e. port 990) is called “FTPS”.

(FTPS) 530 No client certificate presented.

If this FTP server response is seen in the LastErrorText, it means that the SSL/TLS connection requires a client-side certificate with private key.   Prior to connecting, the client-side certificate should be specified by calling the SetSslClientCert, or SetSslClientCertPfx.  Make sure to check the return value of these methods for success/failure.  If the SetSslClientCert* method fails, then no client-side certificate has been set, and the Chilkat SSL/TLS handshake implementation will send a 0-length client-side certificate, which is the correct response for servers that request a client-side cert but don’t actually require it.

If the certificate object passed to SetSslClientCert is a pre-installed certificate on the Windows operating system, then make sure it was installed from the PFX such that the private key is exportable.  Chilkat’s SSL/TLS handshake code needs access to the client-side private key when a client-side cert is required.

FTP Directory Synchronization and Removing Files

Chilkat FTP2 provides methods for synchronizing a local directory tree with a remote directory tree — in both directions.  The SyncLocalTree method downloads files from the remote tree to the local tree, and SyncRemoteTree uploads files from the local tree to the remote tree.  Both have a number of different “modes” that determine which files are transferred, depending on existence, last-modified date comparison, etc.  However, one thing that is intentionally missing is a feature to automatically delete files that do not exist on the other side.  The synchronization is only additive, not deletive.  There is a workaround though, as explained below.

Case #1: Deleting files in the remote tree that do not exist in the local tree.

Determine which files need to be deleted by calling SyncLocalTree using mode #1.  Write an event callback handler for OnBeginDownloadFile (see http://www.chilkatsoft.com/p/p_385.asp ) to skip every download so that nothing is acutally downloaded.  Instead, you’ll build a list of files that would’ve been downloaded because they exist on the server but not locally.  Then write a loop that iterates over the list and calls DeleteRemoteFile for each.

Case #2: Deleting files in the local tree that do not exist in the remote tree.

The same technique is used.  Determinen which files need to be deleted by calling SyncRemoteTree using mode #1.  Write an event callback handler for OnBeginUploadFile to skip every upload so that nothing is actually uploaded.  Instead, you’ll build a list of files that would’ve been uploaded because they exists locally but not on the server.  Then write a loop that iterates over the list can deletes each local file.

SOLVED: 425 Unable to build data connection: Operation not permitted

This error message is specific to the ProFTPd server.  A Chilkat customer found this error within the LastErrorText after trying to upload or download a file, or retrieve a directory listing:

425 Unable to build data connection: Operation not permitted

The latest ProFTPd server has a configuration setting that by default requires SSL/TLS sessions to be re-used, and this breaks *many* FTP clients, including Chilkat. The fix to be made on the server is to add the following line to the proftpd.conf and then restart it.

TLSOptions NoSessionReuseRequired