Damn You Windows Firewall!

One particular cause of the following error is finally known:

sockRecv(47ms):
    WindowsError: An existing connection was forcibly closed by the remote host.
    WindowsErrorCode: 0x2746
    numBytesRequested: 5
    Failed to receive data on the TCP socket
--sockRecv
sockRecv failed.
sockRecvN_buf: Did not receive the exact number of bytes desired.
numBytesToReceive: 5

This error was caused by the Windows Firewall doing stateful FTP filtering. The solution is to disable stateful FTP filtering so that the firewall does not block any FTP traffic. (This is typically prescribed for the server-side of FTP, but it also applies to the client-side.)

Note: The Windows Firewall’s stateful FTP filtering only causes problems when the connection is SSL/TLS. Clear (non-secured) connections are not affected.

To disable stateful FTP filtering, open an administrative command prompt, and type the following:

netsh advfirewall set global StatefulFTP disable

Also Note: The above error could also be a result of the connection truly being closed by the remote host, or something on the server-side. This solution only applies to the case where Windows Firewall (on the client-side computer) is interfering with a SSL/TLS FTP connection.

How to Reproduce the Problem

To reproduce the problem, write a program that establishes a TLS connection with an FTP server.   After authenticating, write a simple loop that calls ftp.Noop 1500 times.   If Windows Firewall is interfering, it is likely to fail in the same place every time.  For example, in my case the error occurs on iteration 699 every time.

Also, the error may not be easily reproducible even if Windows Firewall is a problem.  For example, the problem seemed to be not reproducible if the FTP server was not a Windows-hosted FTP server.   One might suspect the server-side to be the problem, but the connection reset ([RST, ACK] in a WireShark trace) definitely originated from the client-side.

Disabling the firewall’s Stateful Packet Inspection (SPI) feature solved the problem every time. (It should be the case that stateful packet inspection should NEVER try to inspect the packets of a TLS encrypted channel for the simple fact that it’s impossible to inspect.  The firewall does not have the ability to decrypt the packets in the channel, and therefore it cannot inspect anything. The firewall’s only choices SHOULD be to simply allow or disallow the connection.  I don’t understand why stateful packet inspection should ever be “monkeying around” with encrypted channels..)

v9.5.0.58 Update: New Features, Fixes, Changes, etc.

  • Rest: Added the Rest class.
  • Jwt: Added the Jwt class.
  • AuthAws: Added the AuthAws class.
  • AuthGoogle: Added the AuthGoogle class.
  • AuthAzureAD: Added the AuthAzureAD class.
  • AuthAzureStorage: Added the AuthAzureStorage class.
  • ServerSentEvent: Added the ServerSentEvent class.
  • CertStore: Fixed a memory leak in the LoadPemFile method.
  • Charset: Fixed problems in the VerifyData method having to do with us-ascii and utf-8.
  • C++: Now includes utf16_t classes. Except for certain classes such as CkString and CkByteData, there are three options for each C++ class:
    the multibyte class, the wchar_t class, and the utf16_t class. For example, CkSFtp, CkSFtpW, and CkSFtpU.
  • FTP2: (FIXED) The SyncRemoteTree/SyncRemoteTree2 progress monitoring (event callbacks) were not working
  • FTP2: Added the LargeFileUpload method.
  • IMAP: Added new methods: HasCapability, GetQuotaRoot, GetQuota, and SetQuota.
  • LastErrorText: The Windows logged-on username is no longer automatically logged at the top of every LastErrorText.
    (This was originally done to help w/ support issues so tht Chilkat might notice if the program runs in the context of a Service or IIS, which could be a clue to the cause of certain problems.)
  • SSL/TLS: Internal client-side and server-side renegotiate problems were fixed.
  • FTP2: Backslash chars in paths sent to the server are automatically converted to forward-slashes
  • PrivateKey/PublicKey: Deprecated many methods and added new methods. Originally these classes were for RSA keys, but have since become
    a classes that can contain either RSA, DSA, or ECC keys. Methods with “RSA” in the name were deprecated. Also, methods with confusing names were deprecated and replaced with methods having better names.
  • SFTP: Fixed problems with certain SSH/SFTP servers where certain downloads would hang.
  • TLS: Added support for the SHA512 server key exchange hash algorithm. (See http://www.chilkatforum.com/questions/10395/sslallowedciphers-options)
  • StringBuilder: Added a StringBuilder class, which is more helpful in some older programming languages as opposed to others.
  • Email: Fixed problems with comma characters found in email addresses that are Q/B encoded.
  • JsonObject: Crash bug fixed in the StringOf method.
  • Email: (FIXED) Comma characters embedded in an email address friendly name, such as “Doug, Smith ” would interfere with the parsing of the email address.
  • SFTP: Added the SyncDirectives property.
  • LoadTaskResult: (FIXED) The ActiveX LoadTaskResult methods always returned 0, regardless of success/failure.
  • MHT: (FIXED) Chilkat hangs with specific HTML on calling mht.GetEML. The problem had to do with infinite recursion: The CSS that is downloaded itself contained an import for the same CSS (or indirectly contained an import for a CSS that then contained an import for a CSS that was previously downloaded, causing a loop).
  • Crypt2: The value of the KeyLength property should always be in agreement with the number of bytes of key material provided by the application.
    The SetEncodedKey method correctly has the side-effect of also setting the KeyLength property. However, setting the SecretKey property directly did not also have this desired side-effect.
    This was fixed. If the SecretKey property is set with 16 bytes of data (128 bits) then the KeyLength property is automatically updated to 128.
  • PHP: Added Windows builds for PHP as FastCGI with IIS using the Non-Thread Safe (NTS) versions of PHP. See https://www.chilkatsoft.com/php.asp
  • Tar: Fixed a crash bug in the VerifyTar method (for corrupt/garbage input files).
  • CkDateTime: added the GetAsTimestamp and SetFromTimestamp methods.
  • Email: Fixed issues having to do with structuring an email with multipart/related, multipart/alternative, etc.
  • Ftp2: Regarding the ForceIpAddress property: By default, Chilkat will NOT try to bind the data socket to the IP address specified by the ForceIpAddress property.
    You may use a special “bind-” prefix in the ForceIpAddress property to cause the data socket to be bound to the ForceIpAddress property. For example, put_ForceIpAddress(“bind-80.130.210.228”)
    There was an undocumented feature where you can set the ForceIpAddress property equal to the keyword “control”. In this case, the IP address of the control connection is used.
    Also, Chilkat will now recognize the error message “I won’t open a connection to…”, and will automatically retry using the IP address from the message.
  • Csv: (FIXED) Csv.WriteFile2 did not work with utf-16.
  • CkString (C++): Added getUtf16 and setStringUtf16 methods.
  • IMAP/MailMan: (FIXED) The new format of the IMAP-only unlock code unloced IMAP, but not the MailMan object.
  • AbortCurrent: Added the AbortCurrent property to classes that can have event callbacks.
  • HTTP: Fixed a problem involving HTTP redirect responses in combination with HTTP proxies where both SSL/TLS and non-SSL/TLS URLs are involved.
  • IMAP: Added the capability to handle DIGEST-MD5 authentication for IMAP.
  • HTTP: Fixed problems handling 407 responses. Also fixed behavior when receiving a response with a “proxy-connection:close” header.
  • SshKey: (FIXED) SshKey.FromPuttyPrivateKey() failed if comment contained umlauts. See
    http://www.chilkatforum.com/questions/10082/sshkeyfromputtyprivatekey-fails-if-comment-contains-umlauts
  • JsonObject: (FIXED) Calling json.AddObjectAt on an empty JSON document failed.
  • FTP2: Reverted the Microsoft FTP workaround added to v9.5.0.56.
  • Socket: The non-SSL/TLS HTTP proxy functionality was fixed.
  • SCP: The DownloadFile method did not close the channel.

iOS C/C++ Static Library Sizes

There is often alarm at the size of the Chilkat static libraries (.a) for iOS. This should not be of too much concern, because after building your app in Release mode, your app’s executable will NOT grow by the size of the static library. It will only grow by a very small fraction of the total size of the .a

There are some common misunderstandings about static libraries that should be cleared up.

  1. You do NOT include the static library (.a) in the package you submit to the app store.
  2. When building your program, Xcode only pulls in the library code that is directly or indirectly used.   (This is true for any C/C++ linker.)   If, for example, Chilkat added functionality for SNMP, Jabber, and Bitcoin, and if this new code caused growth in the .a by 100MB, then it would only cause an increase in your app’s size if your app actually uses the new code.
  3. You may be unaware of the vast number of already-existing system libs (.a) and the vast sizes if all of these were summed up.  Obviously, these are not included in their entirety in your app’s executable.  The same applies to the Chilkat static libs.
  4. You can always check to see how much your app’s executable actually grows in size after linking with Chilkat by examining the size of the executable after linking.
  5. The Chilkat universal lib is composed of both the simulator static libs (x86 + x86_64), and the device libs (arm64, armv7, armv7s).   Your application built for the App Store would not be including alternatives for the simulator.
  6. This Apple Technical Q&A discusses general techniques for reducing the size of an app:  https://developer.apple.com/library/ios/qa/qa1795/_index.html

iOS: Preparing your Apps for IPv6

Chilkat classes that communicate over TCP/TLS include a property named PreferIpv6. This should be set to YES/true to tell Chilkat to use IPv6 when possible. (If the underlying DNS resolution provides both IPv4 and IPv6 choices, then Chilkat will by default choose the IPv4 address. To tell Chilkat to choose the IPv6 choice, set the PreferIpv6 property equal to YES/true.)

Note: All Chilkat classes that communicate over TCP/TLS will include a PreferIpv6 property. This includes Http, MailMan, Imap, Ftp2, Socket, Ssh, SFtp, etc.

For more information, see Using IPv6 in Chilkat Apps for iOS

Hints for Minimizing Size of Windows CE 6.0 EXE in Visual Studio 2008

Chilkat continues to support (and will continue to support for many years to come) older development environments and platforms. The C++ lib for Windows CE (specifically for VS2008) is one such case. The size of the EXE in this environment can matter. Here is one customer’s helpful hints on how to reduce the size of the EXE when building/linking:

Hints for Minimizing Size of Windows CE 6.0 EXE in Visual Studio 2008

The .NET Assembly “Incorrect Format” Error

If you get this error in a Windows Forms project…

Look in your Project–>Properties (ALT+F7) and go to the “Build” tab.
You’ll see the “Platform target”

– If it’s set to “x64”, then make sure you reference the 64-bit Chilkat assembly.
– If it’s set to “x86”, then make sure you reference the 32-bit Chilkat assembly.
– If it’s set to “Any CPU” and the “Prefer 32-bit” checkbox is checked, then your app will run in 32-bit mode and you should reference the 32-bit Chilkat assembly.
– If it’s set to “Any CPU” and the “Prefer 32-bit” checkbox is NOT checked, then your app will run according to the computer (64-bit or 32-bit). It is unlikely you’re developing on a 32-bit Windows computer, so in that case you’d reference the 64-bit Chilkat assembly.

Chilkat .NET Assemblies – Matching Visual Studio versions to .NET Framework Versions to VC++ Runtime Versions.

 

Each version of Visual Studio has a corresponding version of the .NET Framework that is the natural/default version for it.  It’s the latest version of the .NET Framework that existed when the particular version of Visual Studio was released.  For example:

VS2015 — .NET 4.6
VS2013 — .NET 4.5 (actually 4.5.1, but we only care about major/minor numbers)
VS2012 — .NET 4.5
VS2010 — .NET 4.0
VS2008 — .NET 3.5
VS2005 — .NET 2.0

If using a particular version of Visual Studio, this determines everything else.  (You could, for example, use VS2015 with .NET 4.0, but why bother?)  Here are the choices:

VS2015 .NET 4.6 Chilkat .NET for 4.6 needs VC++ 2015 runtime (also known as VC++ 14)
VS2013 .NET 4.5 Chilkat .NET for 4.5 (built with VS2013) needs VC++ 2013 runtime (also known as VC++ 12)
VS2012 .NET 4.5 Chilkat .NET for 4.5 (built with VS2012) needs VC++ 2012 runtime (also known as VC++ 11)
VS2010 .NET 4.0 Chilkat .NET for 4.0 needs VC++ 2010 runtime (x86)(x64) (also known as VC++ 10)
VS2005/2008 .NET 2.0/3.5 Chilkat .NET for 2.0/3.5 needs VC++ 2005 runtime (x86)(x64) (also known as VC++ 8)

C Language Callbacks

This example demonstrates the general pattern for implementing some standard event callbacks functions for the Chilkat “C” API.  All Chilkat “classes” what have events will use the standard Chilkat events shown in the example below (AbortCheck, PercentDone, and ProgressInfo).

This example demonstrates callbacks for an SFTP download, but the same technique applies to any other Chilkat “class”, such as HTTP, FTP2, MailMan, Rest, etc.

#include <stdio.h>

#include <C_CkSFtp.h>

BOOL myAbortCheck()
    {
    // To abort the current Chilkat method call, return non-zero.
    return 0;
    }

BOOL myPercentDone(int pctDone)
    {
    printf("Percent Done: %d\n",pctDone);

    // To abort the current Chilkat method call, return non-zero.
    return 0;
    }

void myProgressInfo(const char *name, const char *value)
    {
    printf("%s: %s\n",name,value);
    }

void ChilkatSample(void)
    {
    HCkSFtp sftp;
    BOOL success;
    int port;
    const char *hostname;
    const char *handle;

    // Declare event callbacks.
    BOOL (*fnAbortCheck)() = myAbortCheck;
    BOOL (*fnPercentDone)(int pctDone) = myPercentDone;
    void (*fnProgressInfo)(const char *name, const char *value) = myProgressInfo;

    sftp = CkSFtp_Create();

    //  Any string automatically begins a fully-functional 30-day trial.
    success = CkSFtp_UnlockComponent(sftp,"Anything for 30-day trial");
    if (success != TRUE) {
        printf("%s\n",CkSFtp_lastErrorText(sftp));
        CkSFtp_Dispose(sftp);
        return;
    }

    // Setup event callbacks.
    CkSFtp_setAbortCheck(sftp, fnAbortCheck);
    CkSFtp_setPercentDone(sftp, fnPercentDone);
    CkSFtp_setProgressInfo(sftp, fnProgressInfo);

    // Make sure to set a HeartbeatMs for AbortCheck / PercentDone callbacks..
    // Set the HeartbeatMs to 250 milliseconds.
    CkSFtp_putHeartbeatMs(sftp,250);

    //  Set some timeouts, in milliseconds:
    CkSFtp_putConnectTimeoutMs(sftp,5000);
    CkSFtp_putIdleTimeoutMs(sftp,10000);

    //  Connect to the SSH server.
    //  The standard SSH port = 22
    //  The hostname may be a hostname or IP address.

    hostname = "my-Sftp-Server-Domain-Or-IPAddress";
    port = 22;
    success = CkSFtp_Connect(sftp,hostname,port);
    if (success != TRUE) {
        printf("%s\n",CkSFtp_lastErrorText(sftp));
        CkSFtp_Dispose(sftp);
        return;
    }

    //  Authenticate with the SSH server.  Chilkat SFTP supports
    //  both password-based authenication as well as public-key
    //  authentication.  This example uses password authenication.
    success = CkSFtp_AuthenticatePw(sftp,"myLogin","myPassword");
    if (success != TRUE) {
        printf("%s\n",CkSFtp_lastErrorText(sftp));
        CkSFtp_Dispose(sftp);
        return;
    }

    //  After authenticating, the SFTP subsystem must be initialized:
    success = CkSFtp_InitializeSftp(sftp);
    if (success != TRUE) {
        printf("%s\n",CkSFtp_lastErrorText(sftp));
        CkSFtp_Dispose(sftp);
        return;
    }

    //  Open a file on the server:
    handle = CkSFtp_openFile(sftp,"hamlet.xml","readOnly","openExisting");
    if (CkSFtp_getLastMethodSuccess(sftp) != TRUE) {
        printf("%s\n",CkSFtp_lastErrorText(sftp));
        CkSFtp_Dispose(sftp);
        return;
    }

    //  Download the file:
    success = CkSFtp_DownloadFile(sftp,handle,"c:/temp/hamlet.xml");
    if (success != TRUE) {
        printf("%s\n",CkSFtp_lastErrorText(sftp));
        CkSFtp_Dispose(sftp);
        return;
    }

    //  Close the file.
    success = CkSFtp_CloseHandle(sftp,handle);
    if (success != TRUE) {
        printf("%s\n",CkSFtp_lastErrorText(sftp));
        CkSFtp_Dispose(sftp);
        return;
    }

    printf("Success.\n");


    CkSFtp_Dispose(sftp);

    }

int main()
{
  ChilkatSample();
  return 0;
}

v9.5.0.56 Update: New Features, Fixes, Changes, etc.

  • Stream: Added a Stream class to be utilized more heavily in the future.
  • StreamConnector: Added the StreamConnector class for .NET to allow for System.IO.Stream’s to be used with Chilkat.Stream. See http://www.example-code.com/csharp/stream_connector_cs.asp
  • Crypt2: Added the EncryptStream and DecryptStream methods.
  • Compression: Added the CompressStream and DecompressStream methods.
  • JSON: Added the JsonObject and JsonArray classes.
  • Zip: Memory leak fixed.
  • Ruby: Added Ruby 2.3 builds.
  • PrivateKey: Fixed: The LoadPvkFile method was not working.
  • SshTunnel: Fixed: Large downloads are now working.
  • Socket: Crash bug fixed. Calling SendString when a socket was already disconnected would cause a crash in some circumstances.
  • SMTP/MailMan: The SmtpSessionLog is changed to no longer log the 1st 2500 chars of the MIME BODY, but to instead just log the size (number of bytes) of the MIME BODY.
  • Email: Very rare crash bug fixed.
  • HtmlToXml: Now avoids using CDATA when HTML entities are encountered. Instead, HTML entity expressed in decimal (or hex) notation, such as ’ are automatically decoded to the char that it represents.
  • PureBasic: A method that returns a string will return an empty string instead of a 0/NULL. Make sure to use the LastMethodSuccess property to check to see if a string method succeeded or failed.
  • Ftp2: The IsConnected and NumFilesAndDirs properties should be avoided. They were already deprecated prior to this version release, but this is another recommendation to avoid these properties. Instead use the CheckConnection and GetDirCount methods.
  • C API: Added callback function pointers to the “C” API.
  • Imap: Fixed: AppendMail. When a nonexistent mailbox was provided, the method would hang until the ReadTimeout expired. Now it returns immediately.
  • Email: Fixed: If only the FromName property was set (and not the FromAddress property, nor the “From” property, then the FromName was getting lost when generating the MIME, such as by calling GetMime. This was because technically the email address is still incomplete/invalid.
  • MailMan: Added the following methods: Pop3Connect, Pop3Authenticate, and ConnectFailReason. The connect + authenticate has traditionally been a single method call (or something that automatically occurs with the 1st method making a request). The Pop3Connect and Pop3Authenticate methods make it possible for an application to do each step separately, which helps for reporting errors more specificallly.
  • SSH/SFTP: Added the EnableCompression property. An old server was discovered that did not correctly handle compression. Chilkat automatically disables compression when the problematic servers are encountered, but if undiscovered old servers have the same problem, this property can be set to work around the problem.
  • Email: Fixed: Emails that have address of the format “CN=abc/O=abc” are not altered in any way. An email address can have any form (not just the standard name@address form).
  • Http: Added the ConnectFailReason property.
  • SshTunnel: Fixed: Problems where the data flow in a tunnel became stuck.
  • Email: Workaround for uuencoded email bodies where the uuencoding starts with a “begin …” line.
  • Encoding: Added the “base64url” encoding to the list of supported encodings. See http://cknotes.com/chilkat-binary-encoding-list/
  • HttpRequest: Fixed: The AddParam method stripped bare CR’s from the value. This does not happen anymore.
  • GitHub: The SMTPQ source code is released to GitHub, along with other example projects, distributions, etc. See https://github.com/chilkatsoft
  • SFTP: Fixed: The ReadDirMustMatch and ReadDirMustNotMatch properties did not work correctly.
  • Crypt2, Mime, Email: The default value of the Pkcs7CryptAlg was changed from “RC2” to “AES”.
  • POP3: Added internal workarounds for older POP3 servers that do not support the UIDL command. (It is extremely rare to find a POP3 server that does not support UIDL.)
  • SMTPQ/MailMan: The Windows-only SMTPQ related methods were fixed. The SMTPQ methods were working correctly in v9.5.0.54 and earlier, but a problem was inadvertently introduced in v9.5.0.55.
  • Unicode C++: Fixed: The LastMethodSuccess property was not getting set.
  • Mime: Fixed: Removed “email address cleaning” from CkMime. The MIME header was run through email-specific processing logic to clean/ensure well formatted email addresses. This should not have been done for the general-purpose MIME API.
  • Email: Fixed a SetHtmlBody problem. See http://www.chilkatforum.com/questions/9536/change-html-body-in-a-multipartrelated-mail
  • HttpRequest: Added the StreamChunkFromFile method.