Amazon S3 Problem Resolution Hints

This post describes two problems and solutions for Amazon S3 that were recently encountered:

(1) A Chilkat user found that the 1st HTTP request had a delay of 5 seconds, and then subsequent requests had no delay.  Here’s what was found (reproduced with permission from the Chilkat user):

I thought you might be interested to know that the long lookup times (which were variable) I was seeing when using Chilkat.Http – http.GetHead(url) to check the existence of images in an AWS S3 bucket, were caused by a misconfigured DNS server in our Amazon VPC. The first request was often being sent off to find the DNS in the wrong place, which it did in a circular fashion taking about 4 seconds, then the cached response was used for the next requests.

Took a while to figure out with the help of Amazon support.

In general — if a problem is encountered with any Internet communications where the 1st request takes much longer than subsequent requests — suspect a DNS problem.

(2) A Chilkat user was trying to upload to S3 a 700MB file and encountered this error:

                  sendFile(47ms):
                    filePath: …
                    processAlert:
                      TlsAlert:
                        level: warning
                        descrip: close notify
                      --TlsAlert
                    --processAlert
                    Received close-notify.
                    Did not receive additional application data.
                    Already received close-notify.
                    terminateConnection:
                      TCP connection cleanly closed by peer.
                      Cleanly terminated TCP connection.
                    --terminateConnection
                    Failed to write bytes.
                    Failed to write data to output.
                  --sendFile

Cause and Solution:  HTTP connections can stay open for multiple requests.  If an HTTP response does not include the “Connection: close” header, then Chilkat will keep the connection open and use it for subsequent requests (including S3).  However, if enough time passes before the next request, the HTTP server may choose to disconnect the idle connection.   This is what happened.

In this case, the call to Http.S3_Upload was not the first request.  The connection to the S3 server was open when S3_Upload was called.  However, S3 uploads require a SHA256 hash of the data being uploaded.  To SHA256 hash 700MB (in this case) took a little less then 30 seconds, and this delay was enough for the server to decide to disconnect because the connection was idle.  This causes the above error.

The solution is to call Http.CloseAllConnections prior to uploading a large file.

Chilkat will be adding an internal feature for the next version to track the connection idle time and automatically discard/recover the connection to avoid this pitfall.