Damn You Windows Firewall!

One particular cause of the following error is finally known:

sockRecv(47ms):
    WindowsError: An existing connection was forcibly closed by the remote host.
    WindowsErrorCode: 0x2746
    numBytesRequested: 5
    Failed to receive data on the TCP socket
--sockRecv
sockRecv failed.
sockRecvN_buf: Did not receive the exact number of bytes desired.
numBytesToReceive: 5

This error was caused by the Windows Firewall doing stateful FTP filtering. The solution is to disable stateful FTP filtering so that the firewall does not block any FTP traffic. (This is typically prescribed for the server-side of FTP, but it also applies to the client-side.)

Note: The Windows Firewall’s stateful FTP filtering only causes problems when the connection is SSL/TLS. Clear (non-secured) connections are not affected.

To disable stateful FTP filtering, open an administrative command prompt, and type the following:

netsh advfirewall set global StatefulFTP disable

Also Note: The above error could also be a result of the connection truly being closed by the remote host, or something on the server-side. This solution only applies to the case where Windows Firewall (on the client-side computer) is interfering with a SSL/TLS FTP connection.

How to Reproduce the Problem

To reproduce the problem, write a program that establishes a TLS connection with an FTP server.   After authenticating, write a simple loop that calls ftp.Noop 1500 times.   If Windows Firewall is interfering, it is likely to fail in the same place every time.  For example, in my case the error occurs on iteration 699 every time.

Also, the error may not be easily reproducible even if Windows Firewall is a problem.  For example, the problem seemed to be not reproducible if the FTP server was not a Windows-hosted FTP server.   One might suspect the server-side to be the problem, but the connection reset ([RST, ACK] in a WireShark trace) definitely originated from the client-side.

Disabling the firewall’s Stateful Packet Inspection (SPI) feature solved the problem every time. (It should be the case that stateful packet inspection should NEVER try to inspect the packets of a TLS encrypted channel for the simple fact that it’s impossible to inspect.  The firewall does not have the ability to decrypt the packets in the channel, and therefore it cannot inspect anything. The firewall’s only choices SHOULD be to simply allow or disallow the connection.  I don’t understand why stateful packet inspection should ever be “monkeying around” with encrypted channels..)

v9.4.1.43 Fixed FTP2 XCRC Problem

Fixed an FTP2 XCRC issue. If the AutoXcrc property was turned on, Chilkat FTP2 was not computing the CRC of the local file (or data) uploaded to the server. This caused the CRC comparison to fail after the XCRC command was issued.

Note: This fix only applies when using AutoXcrc in conjunction with a server that actually supports XCRC.

This is a pre-release fix. Pre-release builds are made available upon request (support@chilkatsoft.com). Please make sure to specify the exact build needed (programming language, operating system, architecture, .NET Framework, Visual Studio Version, or anything else that may be required to know exactly which build is required).

ListPattern Property Setting can Affect how FTP2 Gets Directory Information

If an FTP server supports the RFC 3659 extension, which includes the MLSD command, then it is automatically known to Chilkat FTP2 because the FTP server’s response to the FEAT command will include MLSD in the list. (Chilkat automatically sends a FEAT command after connecting so that it knows the capabilities of the FTP server.)

When Chilkat FTP2 recognizes that MLSD is supported, it will use this extension whenever possible. The reason is that MLSD returns more accurate date/time and size information. The drawback is that it can only return a full directory listing. Therefore, if the ListPattern property is set to anything other than “*”, such as “*.txt”, then MLSD cannot be used.

When MLSD is not used, Chilkat must use the FTP protocol’s LIST command which returns the directory listing in a human-readable format. The problem with the LIST response is that different operating systems and FTP servers respond with different listing formats. Chilkat FTP2 automatically recognizes and parses all known directory formats, including formats for uncommon systems such as VAX/VMS, MVS, AS/400, etc. Another issue with the human-readable response is that date/time information may be inaccurate (for example, timezone may not be known, or for older files the HH:MM:SS may not be provided).

Therefore, when using FTP2 to examine the contents of a remote directory, if accurate date/time information is important, then make sure the ListPattern is set to “*” and write your application code to iterate over all the files in the remote directory, perhaps only acting on the files that have names matching what your app needs.

Of course, even when using “*”, if the FTP server itself does not support MLSD, then your app is at the mercy of whatever information is or is not included in the human readable directory listing returned by the LIST command.

SFTP vs. FTPS

Clarification on the acronyms “SFTP” and “FTPS”

“SFTP” is the Secure File Transfer Protocol over SSH.  It is a protocol unrelated to the FTP protocol. (It is actually a subsystem of SSH.)  The Chilkat SSH / SFTP component is used for “SFTP”.  SFTP is achieved by connecting to an SSH server at port 22.

On the other hand, the Chilkat FTP2 component is for FTP.   FTP servers listen at port 21 (non-SSL/TLS) and port 990 (SSL).  FTP over SSL (i.e. port 990) is called “FTPS”.

(FTPS) 530 No client certificate presented.

If this FTP server response is seen in the LastErrorText, it means that the SSL/TLS connection requires a client-side certificate with private key.   Prior to connecting, the client-side certificate should be specified by calling the SetSslClientCert, or SetSslClientCertPfx.  Make sure to check the return value of these methods for success/failure.  If the SetSslClientCert* method fails, then no client-side certificate has been set, and the Chilkat SSL/TLS handshake implementation will send a 0-length client-side certificate, which is the correct response for servers that request a client-side cert but don’t actually require it.

If the certificate object passed to SetSslClientCert is a pre-installed certificate on the Windows operating system, then make sure it was installed from the PFX such that the private key is exportable.  Chilkat’s SSL/TLS handshake code needs access to the client-side private key when a client-side cert is required.

FTP Directory Synchronization and Removing Files

Chilkat FTP2 provides methods for synchronizing a local directory tree with a remote directory tree — in both directions.  The SyncLocalTree method downloads files from the remote tree to the local tree, and SyncRemoteTree uploads files from the local tree to the remote tree.  Both have a number of different “modes” that determine which files are transferred, depending on existence, last-modified date comparison, etc.  However, one thing that is intentionally missing is a feature to automatically delete files that do not exist on the other side.  The synchronization is only additive, not deletive.  There is a workaround though, as explained below.

Case #1: Deleting files in the remote tree that do not exist in the local tree.

Determine which files need to be deleted by calling SyncLocalTree using mode #1.  Write an event callback handler for OnBeginDownloadFile (see http://www.chilkatsoft.com/p/p_385.asp ) to skip every download so that nothing is acutally downloaded.  Instead, you’ll build a list of files that would’ve been downloaded because they exist on the server but not locally.  Then write a loop that iterates over the list and calls DeleteRemoteFile for each.

Case #2: Deleting files in the local tree that do not exist in the remote tree.

The same technique is used.  Determinen which files need to be deleted by calling SyncRemoteTree using mode #1.  Write an event callback handler for OnBeginUploadFile to skip every upload so that nothing is actually uploaded.  Instead, you’ll build a list of files that would’ve been uploaded because they exists locally but not on the server.  Then write a loop that iterates over the list can deletes each local file.

SOLVED: 425 Unable to build data connection: Operation not permitted

This error message is specific to the ProFTPd server.  A Chilkat customer found this error within the LastErrorText after trying to upload or download a file, or retrieve a directory listing:

425 Unable to build data connection: Operation not permitted

The latest ProFTPd server has a configuration setting that by default requires SSL/TLS sessions to be re-used, and this breaks *many* FTP clients, including Chilkat. The fix to be made on the server is to add the following line to the proftpd.conf and then restart it.

TLSOptions NoSessionReuseRequired

Handling Accented Characters in Filenames w/ IIS 7.5 FTP

To handle accented characters in filenames correctly, the FTP client must know what character encoding is used to represent these characters (such as utf-8, iso-8859-1, etc.). The Ftp2.DirListingCharset property indicates the character encoding to be used when interpreting the bytes of a filename (in a directory listing). By default, it is set to “ANSI”, which is another name for the default multi-byte (non-Unicode) charset of the computer. When connecting to an FTP server, if the server indicates it supports the “UTF8” feature, the DirListingCharset property will be implicitly set to “utf-8”. (You may check the value of the DirListingCharset after connecting to see what it contains.)

There has been some trouble with IIS 7.5 FTP.

It turns out that IIS 7.5 FTP has a new UTF 8 advanced option and when I set this to false, all accented characters appears correctly if the DirListingCharset remains at it’s default value.

In this case, it must be that the IIS 7.5 FTP server does not indicate it supports “UTF8” feature, and the DirListingCharset remains at its default  value of “ANSI” and the FTP server sends ANSI characters.  Since both are in agreement, there is no problem.

If this option is set to utf8, whether we set DislistingCharset or not, we don’t get the accented characters.

It is probable that that IIS 7.5 FTP still did not indicate that it supports the “UTF8” feature, and after connecting the DirListingCharset is implicitly set to ANSI.  The solution is to force the DirListingCharset to “utf-8” by setting this property after connecting.