HMAC Hex Key Ambiguity

This happens all the time..

Quite often, a service provider will provide instructions for HMAC generation, and will provide a hexadecimal HMAC key in the example, such as:

01A0251D601FEF3305A38B124068A001CF3F099AA187DB2886314C67CDFDEBCD

This is implicitly ambiguous because there are two ways to interpret the instructions:

1) The HMAC key is composed of the ascii bytes ‘0’, ‘1’, ‘A’, ‘0’, ‘2’, ‘5’, etc. In other words, the HMAC key is 0x30, 0x31, 0x41, etc.

or

2) The HMAC key is composed of the bytes represented by the hex string. In other words, the HMAC key is 0x01, 0xA0, 0x25, etc.

Both can be accomplished using Chilkat.
For the #1 case, one would call

crypt.SetMacKeyString("01A0251D … 67CDFDEBCD")

For the #2 case, one would call

crypt.SetMacKeyEncoded("01A0251D … 67CDFDEBCD","hex")

POP3 Error: No X-UIDL header found

The Chilkat MailMan class can fetch emails from a POP3 server in two ways: by sequence number, or by UIDL. When an email is fetched by UIDL, or fetched in a way such that a full mapping of UIDL’s to sequence numbers was retrieved, Chilkat will add an “X-UIDL” header to the Email object that is returned. This allows for the email object to be uniquely associated with the email on the server. (Sequence numbers change with each POP3 session, but UIDL’s don’t.)

For convenience, an email object is passed to some MailMan methods, such as w/ DeleteEmail. If the email was not retrieved in a way such that the UIDL was known, then the X-UIDL header will not be present, and the MailMan has no way of specifying which email on the server to delete. For example, if the email was downloaded by calling MailMan.FetchByMsgnum, then no UIDL was ever known (unless perhaps MailMan.GetUidls was previously called in the same POP3 session). Thus the error message in the LastErrorText is “No X-UIDL header found”.

One solution is to call MailMan.GetUidls beforehand. One call at the start of the POP3 session is sufficient.

SSH/SFTP Error: Must first connect to the SSH server

The following error is explained in this post:

ChilkatLog:
  DownloadFileByName:
    DllDate: Apr 25 2018
    ChilkatVersion: 9.5.0.73
    UnlockPrefix: *
    Architecture: Little Endian; 64-bit
    Language: Cocoa Objective-C
    VerboseLogging: 0
    SftpVersion: 3
    Component successfully unlocked using purchased unlock code.
    Must first connect to the SSH server.
  --DownloadFileByName
--ChilkatLog

The above error can happen after a long period of inactivity. Let’s say your application successfully connected and authenticated w/ the SFTP server, did some things, and then did not do anything else for a long period of time.  Meanwhile, the SFTP server decides to disconnect because the client has been inactive for too long.  The client (your app + Chilkat) would only discover that the server has dropped the connection once it tries to do something, such as in a call to DownloadFileByName.  The non-connected socket is discovered in the 1st attempt to send a message, and thus you receive the above error.

There are two possible actions an application might take:

  1. Prevent the disconnect by periodically calling sftp.SendIgnore to keep the connection from being inactive.
  2. (auto-recovery)  If DownloadFileByName (or some other method) returns false/0 to indicate failure, examine the sftp.IsConnected property.  If not connected, then automatically re-connect, re-authenticate, and call InitializeSftp to get back to a connected state, and then retry the method.

Chilkat v9.5.0.73 Release Notes

The v9.5.0.72 release notes are available here: Chilkat v9.5.0.72 Release Notes

v9.5.0.73 Release Notes:

  • Email The SetHtmlBody method, in certain circumstances, would incorrectly set the top-level MIME header to text/html for multipart messages. This was fixed.
  • HTTP Non us-ascii chars in the URL path are now always URL encoded using the utf-8 encoding.
  • OAuth2 Added the UseBasicAuth and AppCallbackUrl properties.
  • MailMan Added methods SendMimeBd and FetchMimeBd.
  • Bounce Fixed a few situations where bounce type 11 (Suspected Bounce) was returned, but should have been 6 (Auto-Reply). In general, minor improvements to Bounce categorization are added as customers report issues, and each new Chilkat version can be assumed to have minor Bounce categorization refinements.
  • Tar Fixed crash (access violation) in the Untar method. This was caused by a corrupt tar file where the internal header contained garbage bytes.
  • Objective-C/CkoJsonObject In iPhoneOS11.2.sdk/usr/include/complex.h, we find the following macro: “#define I _Complex_I”. This caused a compile error because the CkoJsonObject class has a property named “I”. Chilkat updated CkoJsonObject.h to “#undef I” to avoid the conflict.
  • Electron Added builds for Electron 1.8. However the package naming triggered some npm spam detectors (false positives of course), and Chilkat is working to get it resolved. Chilkat will produce builds for Electron 2.0 in the near future.
  • CkByteData Added a SecureClear bool property (get_SecureClear, put_SecureClear). If set to true, then whenever the internal data is deallocated, the memory is first overwritten with 0 bytes.
  • MIME Binary MIME with null bytes in some bodies became corrupted because 0 bytes were replaced with SPACE chars. This is fixed.
  • Signed/Encrypted Email Fixed: The “micalg” attribute incorrectly remained in the Content-Type header field for the encrypted MIME part when the email is also signed.
  • REST Fixed: Stream sources used for uploading were not properly closed after the upload finished.
  • C++ Builder/CkAuthAzureStorage Fixed: The x-ms-date header did not automatically get the correct current date/time for HTTP requests. This only happened for C++ Builder and Delphi builds of Chilkat.
  • Email Added flexibility in parsing non-compliant RFC822 date strings, where the month name and month day number are not in the correct order as specified by RFC822.
  • MIME parsing (general) Added more internal flexibility for handling mixtures of CRLF and bare-LF line endings.
  • XmlDSigGen Now capable of using non-exportable private keys on Windows, such as for A3 certificates where the private key is on a hardware token.
  • SSH Fixed a rarely encountered “handshake” problem.
  • HTTP Changed the default Content-Type for the PostJson method to be “application/json”. The original default value, “application/jsonrequest” was the initial “standard” years ago, but seems to never be used nowadays. If an “application/jsonrequest” is needed (and I doubt it will ever be needed), then PostJson2 may be called to explicitly specify the Content-Type.
  • HTTP The default value of the S3Ssl property is now true.
  • Zip Fixed rare problems involved with rewriting zip archives, when the “move from temp zip to target” fails.
  • Email Fixed certain automatic MIME structuring issues w.r.t. multipart/alternative and multipart/related.
  • Compression Added the DeflateLevel property to the Compression class.
  • CkString The removeDelimited method was missing for Ruby, Java, Perl, Python, Tcl, and PHP.
  • SFTP Fixed: The SyncTreeDownload method was not firing the DownloadRate callback.
  • PrivateKey Fixed: The GetPkcs8Pem method (for ECC keys) was returning PKCS1 but should’ve been returning PKCS8.
  • HTTP Added the SharePointOnlineAuth method.
  • HTTP Fixed problems with the S3_GenerateUrl and S3_GenerateUrlV4 methods.
  • SSH Fixed: After doing a ReadDir, the SFtpFile.IsDirectory property was not correct for some types of SSH servers.
  • Zip Added the PwdProtCharset property.
  • SOCKS5 Fixed problems with IPv6 addresses when using SOCKS5 proxies.
  • Compression Added the CompressSb and DecompressSb methods.
  • JSON Added the methods DtOf and DateOf to both JsonObject and JsonArray.
  • HTTP Fixed problems with non-us-ascii chars in URLs for downloads.
  • StringBuilder Added the ReplaceAfterFinal method.
  • HTTP Fixed: The LastHeader property was empty after the PostUrlEncoded method.
  • SSH Fixed slowness for SSH commands the emit a large amount of output.
  • HTTP Fixed rare server certificate verification problem when the server provides out-of-order certificates in the TLS handshake.
  • FTP2 Fixed: The IdleTimeoutMs property was not being honored for DNS problems.
  • StringBuilder Added the WriteFileIfModified method.
  • PureBasic Fixed a compile error in the CkHttp.pb file.
  • SSH/SFTP/SshTunnel Fixed a host key signature verification failure for certain situations.
  • SSH/SFTP/SshTunnel Added the UncommonOptions property, which will be a place to specify future unforeseen workarounds that may be required for particular SSH servers (old or new).
  • Rest/Socket Fixed a problem when a non-standard HTTP port (not 80 nor 443) is used in for the original Socket connection in conjunction with the Rest.UseConnection method.
  • RSA Added the SetX509Cert method to make it easy to use the private key of certificate. Also allows for A3 certificates where the private key is non-exportable (on a Windows system) such as on a hardware token.
  • XmlDSigGen Added the “X509Data+KeyValue” option for the KeyInfoType property.
  • JsonArray Added the FindString and FindObject methods.
  • Zip Fixed reliability issues in the UnzipToStream method.
  • XmlDSigGen Fixed: The X509SerialNumber in the X509IssuerSerial needed to be in decimal, not hex.
  • S/MIME Fixed an extremely rare issue where a digital signature verification failed but should’ve been successful.
  • JsonObject Fixed the “Unable to lock my JSON object.” error that would be returned if SetStringOf was called on an empty JsonObject.
  • CkString The loadFile method will now recognize Unicode/utf-8 BOMs and will load files correctly based on the BOM encountered.

Windows 10 1803 can’t run EXE files from a network shared folders

Chilkat has been receiving support email with the following error:

          connectSocket:
            connect_ipv6_or_ipv4:
              resolveHostname6:
                getAddressInfo:
                  Failed to get host address info. (3)
                  SocketError: Error 0x2afb
                  Check to make sure the connection is not blocked by a firewall or anti-virus port filtering.
                  hostOrIpAddr: ****
                  port: 
                  See http://www.chilkatforum.com/questions/11627/sftp-failed-to-get-address-info for a possible cause of this error.
                  Versions of Windows earlier than Windows XP are limited to handling IPv4 only
                  On Windows Server 2003 and Windows XP, IPv6 addresses are returned only if IPv6 is installed on the local computer.
                --getAddressInfo
              --resolveHostname6
              Domain to IP address resolution failed.
            --connect_ipv6_or_ipv4
          --connectSocket

One user described the situation perfectly:

This issue I am having is with FTP2 and the Email components. I have enclosed the error screen for the FTP.

  1. I did a windows update last night. It has worked fine for 2 years until the update.
  2. There is no issue if I run the program in the VB6 interpreter. It only happens when I run the .exe program.
  3. If I run the .exe program from the local drive, it works fine. It only happens when I run it from the network drive.
  4. I have another computer set up and it works fine from the network drive.

Please let me know if you have an idea. I am running windows 10 (32bit)


The answer is found here.

In summary: “It can be concluded that Windows 10 update 1803 for security reasons does not allow you to open network connections in programs running from shared folders that are accessible only using the SMBv1 protocol. As network folders, you need to use devices that support SMBv2 or SMBv3.”

See http://woshub.com/windows-10-1803-cant-run-exe-files-shared-folders/ for more details.

Yield and SleepMs in PowerBuilder

This is a note for PowerBuilder programmers: Some Chilkat classes provide a SleepMs method, which is provided as a convenience. The SleepMs method puts the thread to sleep for a number of milliseconds. However, this is not the same as a PowerBuilder Yield, which “Yields control to other graphic objects, including objects that are not PowerBuilder objects. Yield checks the message queue and if there are messages in the queue, it pulls them from the queue.”

The above was found by a Chilkat customer w/ regard to the Task.SleepMs method:

The problem was the when we were calling awParent.Dynamic wf_download_percent(percent) to display the download percentage on a progress window of ours, the window wasn’t always getting control to show that! I needed to add a Yield() right after that and then it was perfect. That’s a fairly common problem in PowerBuilder. I have to admit I would have thought that your SleepMs method would be tantamount to a Yield(), but obviously it’s not.

Perhaps you might consider adding something to your PowerBuilder samples that show progress of Async actions, where instead of just logging progress it was calling a made-up function to display progress, and then add a Yield() after that to make it clear that is needed.

Chilkat v9.5.0.72 Release Notes

The v9.5.0.71 release notes are available here: Chilkat v9.5.0.71 Release Notes

v9.5.0.72 Release Notes:

  • FTP2 Fixed FTP implicit SSL/TLS uploads for some FTP servers.
  • CkString Fixed: In the Delphi DLL build, the CkString.split* and tokenize* methods caused a crash.
  • Encryption Modification: Encrypting 0 bytes for block cipher algorithms with padding now results in one block of output, rather than 0 bytes.
    (Decrypting the 16-byte block will return the original 0 bytes.) This only applies to block ciphers operating in modes such as ECB, CBC, etc. This does not apply to stream ciphers, or block ciphers operating in a streaming mode.
  • XML Fixed cases where “ ” was getting changed to 
 in round-trip load/save.
  • HTTP The Http.AwsSignatureVersion property default value is changed from 2 to 4. All AWS regions support v4 signatures, whereas S3 regions deployed after January, 2014 do not support V2.
  • PrivateKey Fixed a problem in loading certain (seldom encountered) types of private keys. Specifically, keys using PBES1/RC4.
  • PKCS7 Signature Fixed certain cases where creating PKCS7 signatures embedded the same certificate twice.

C# TreeView to JSON and Back

Here’s a C# class for persisting a TreeView to JSON, and for restoring a TreeView from JSON.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;

namespace ChilkatRelease
    {
    public class TreeviewPersist
        {

        // Persist the TreeView to a JSON string.
        static public string ToJson(TreeView treeView)
            {
            Chilkat.JsonObject tvJson = new Chilkat.JsonObject();
            Chilkat.JsonArray tvNodes = tvJson.AppendArray("treeViewNodes");

            TreeNodeCollection nodes = treeView.Nodes;
            foreach (TreeNode n in nodes)
                {
                serializeTree(tvNodes, n);
                }

            tvJson.EmitCompact = false;
            return tvJson.Emit();
            }

        // Clears the passed-in treeView and rebuilds from JSON.
        static public void FromJson(string strJson, TreeView treeView)
            {
            treeView.Nodes.Clear();

            Chilkat.JsonObject tvJson = new Chilkat.JsonObject();
            tvJson.Load(strJson);
            Chilkat.JsonArray tvNodes = tvJson.ArrayOf("treeViewNodes");

            int numNodes = tvNodes.Size;
            for (int i = 0; i < numNodes; i++)
                {
                Chilkat.JsonObject json = tvNodes.ObjectAt(i);

                if (json.IsNullOf("parentName"))
                    {
                    TreeNode node = treeView.Nodes.Add(json.StringOf("name"), json.StringOf("text"));
                    restoreNode(node, json);
                    }
                else
                    {
                    // Assumes unique names (i.e. keys)
                    TreeNode[] foundNodes = treeView.Nodes.Find(json.StringOf("parentName"), true);
                    if (foundNodes.Length > 0)
                        {
                        TreeNode node = foundNodes[0].Nodes.Add(json.StringOf("name"), json.StringOf("text"));
                        restoreNode(node, json);
                        }
                    }

                }
            }

        // Restore the properties of a TreeNode from JSON.
        static private void restoreNode(TreeNode node, Chilkat.JsonObject json)
            {
            node.Tag = json.StringOf("tag");
            node.Text = json.StringOf("text");
            node.ToolTipText = json.StringOf("toolTipText");
            node.Checked = json.BoolOf("checked");
            }

        // Recursive method to add TreeView nodes to the JSON.
        static private void serializeTree(Chilkat.JsonArray tvNodes, TreeNode treeNode)
            {
            tvNodes.AddObjectAt(-1);

            Chilkat.JsonObject json = tvNodes.ObjectAt(tvNodes.Size-1);
            json.UpdateString("name", treeNode.Name);

            TreeNode parent = treeNode.Parent;
            if (parent != null)
                {
                json.UpdateString("parentName", treeNode.Parent.Name);
                }
            else
                {
                json.UpdateNull("parentName");
                }

            json.UpdateString("tag", (string)treeNode.Tag);
            json.UpdateString("text", treeNode.Text);
            json.UpdateString("toolTipText", treeNode.ToolTipText);
            json.UpdateBool("checked", treeNode.Checked);

            foreach (TreeNode tn in treeNode.Nodes)
                {
                serializeTree(tvNodes,tn);
                }  

            }


        }
    }

Chilkat v9.5.0.71 Release Notes

The v9.5.0.70 release notes are available here: Chilkat v9.5.0.70 Release Notes

v9.5.0.71 Release Notes:

  • FileAccess Added the GetFileTime method to get the last-modified, created, or last-access file date/time.
  • Email Automatically fixes FROM email addresses to prevent email spoofing as described at https://www.mailsploit.com/index
  • FileAcccess If FileSize() returns -1 (failed), then the LastMethodSuccess property was not properly set. (The LastMethodSuccess is a property common to Chilkat classes that is automatically set when any method that returns an integer, string, or object is called.)
  • WebSocket Fixed so that if corrupt data is received (i.e. bytes that do not conform to the WebSocket framing protocol), then the connection is closed as per the WebSocket standard.
  • OAEP Added the OapeMgfHash property to the Email, Crypt2, Mime, and Rsa classes.
  • XmlDSigGen Added support for XML canonicalization WithComments algorithms.
  • StreamConnector/Crypt2 Fixed Crypt2.DecryptStream for cases in .NET when a StreamConnector is used.
  • AuthAws Fixed AWS authentication for Amazon SES.
  • Jwe Fixed problem with RSA-OAEP-256. Also added support for RSA-OAEP-384 and RSA-OAEP-512
  • Rest Added the ConnectTimeoutMs property.
  • MailMan Added the LogMailReceivedFilename property to log a POP3 conversation to a file in real-time.
  • PHP Added thread-safe builds for PHP 7.2 (as opposed to only supporting NTS builds).
  • SFTP Added the XferByteCount property to allow for applications to access the current upload or download transfer byte count for ongoing asynchronous transfers.
  • SSH/SFTP Added the ServerIdentifier property.
  • Csv Added property EnableQuotes such if set to False/0, causes the parser to not treat double-quotes chars as special.
  • C++ Builder Added a build for C++ Builder 10, which includes a build for the 32-bit clang library.
  • Jwe Fixed to internally restrict IV’s to 96 bits as standards dictate.
  • Node.js Added builds for Node.js 9.*.*
  • Dkim Added methods that use BinData. Deprecated methods that use byte arrays or CkByteData.
  • StringBuilder Added methods for punycode.
  • Ruby Added builds for Ruby 2.5.*
  • OpenSSL Decrypt Added feature to be able to decrypt files encrypted using an openssl command such as “openssl enc -e -aes-256-cbc -in hamlet.xml -out hamlet.enc -pass file:./secret.txt”. See https://www.example-code.com/csharp/openssl_enc_decrypt.asp
  • CkHashtable Fixed crash bug in the AddFromXmlSb method.
  • Email Added a method named ApplyFixups. (Read more about it in the online reference documentation.)
  • SshKey Added support for the EC key format used by openssl. (No coding changes required, Chilkat automatically recognizes the key format when loading/parsing.)
  • FileAccess Fixed issues with TreeDelete: (1) It returned false when it actually succeeded, and (2) it failed to delete read-only files.
  • SecureString Added the SecureString class to help keep passwords encrypted in memory.
  • Crypt2 Added the DecryptSecureENC and EncryptSecureENC methods.
  • Imap Added the LoginSecure method.
  • MailMan Added the SetPassword method (to set the password using a SecureString rather than just setting the Password property).
  • Ssh, SFtp, SshTunnel Added AuthenticateSecPw and AuthenticateSecPwPk methods.
  • Http Added the SetPassword method so that a password may be set via a SecureString.
  • Rest Added the SetAuthBasicSecure method.
  • SFtp Added the following methods: SymLink,HardLink, ReadLink, and FSync.

Minimizing Time to Send Email

Programmers using Chilkat to send email can do some simple things, and can avoid common mistakes to optimize the time it takes to send an email.

1) A common mistake is when a programmer calls mailman.VerifySmtpConnection and mailman.VerifySmtpLogin prior to actually sending the email. The intent of the Verify methods is for an application to examine connectivity and login success after a call to SendEmail returns a non-success status. The best option is to simply call SendEmail first. If successful, then all is good. If not, then first examine the mailman.SmtpFailReason property. It will provide a general reason for failure, and two of these reasons are ConnectFailed and AuthFailure.

Many mail servers are SLOW in the time it takes to respond to a login. For example, we’ve seen many cases where Office365 takes 3-4 seconds to respond to the AUTH command (i.e. the login). An application wouldn’t want to experience this delay twice for every email sent. This is what happens if VerifySmtpLogin is called prior to each call to SendEmail.

2) When SendEmail completes successfully, Chilkat leaves the authenticated connection with the SMTP server open so that a subsequent call to SendEmail will use the already-established connection. However, if the connection is idle for a long time, a mail server is likely to close the connection. This is no problem for Chilkat because if SendEmail finds the connection closed, it will automatically re-establish and re-authenticate and all is good — except for the time delay in performing the re-connect and re-authenticate, which can be large. Therefore, an application could periodically call mailman.SmtpNoop to prevent the connection from being idle for too long, and thus prevent the mail server from closing the connection. This way, it’s more likely that the next SendEmail will find a valid/authenticated connection.

3) The LastErrorText property is common to most Chilkat objects. A feature of the LastErrorText is that if VerboseLogging is turned on, then the elapsed time (in milliseconds) for each context is shown within LastErrorText. (For cases where a context takes less that 1 milliseconds, the elapsed time is omitted.) Thus an examination of the verbose mailman.LastErrorText can help identify where the time was spent in sending an email. One can identify the time spent in the TLS handshake, in the SMTP authentication, and in the sending of the DATA and waiting for the final SMTP status response.

4) Finally, if you’re using an old version of Chilkat (meaning years old), make sure to update to the latest version. Over the years Chilkat is always improving the internals for performance and memory usage. At this point in time, slowness in sending email is highly likely to be caused by server slowness or other external factors, and not something within Chilkat. (I would venture to say this is also the case for any client-side implementation of SMTP — the slowness is not likely to be caused by the client-side implementation.)