Working with Huge (Extremely Large) JSON Files

Question:

I am using the following calls in Visual FoxPro with Chilkat_9_5_0.JsonObject (9.5.0.80) and the resulting JSON is writing only {}

oChilkatJson=Createobject('Chilkat_9_5_0.JsonObject')
oChilkatJson.LoadFile("pathToLargeJSONFile")
oChilkatJson.EmitCompact=1
=STRTOFILE(oChilkatJson.Emit(),pathToOutputFile)

Original Size of JSON: 259 MB (271,982,395 bytes)

New JSON outputs as: {}

I could use the https://www.example-code.com/foxpro/create_complex_json.asp​ to read and write each JSON property/object to memory and perform the one EMIT() a the end, I would think it would be the same result.

It reads it fine and I can loop through the JSON object.

Is there a file size limitation to writing using EMIT()?

What method is the best way to read and write large JSON.  The JSON files we work with could be larger than this, up to 2GB.

Answer:

This is a great question because the answer allows me to describe a general situation that should be avoided.   In some programming languages, it’s generally not a good idea to pass huge amounts of data to a method, or return huge amounts of data from a method.   Programming languages that uses the ActiveX, such as FoxPro, VB6, DataFlex, VBScript, Classic ASP, etc. would qualify.   When an ActiveX method returns a string, it must marshal the string as a BSTR (utf-16) and return it to the application.  This is a more-than-doubling of the memory required because the utf-8 held internally by the JSON object is mostly or entirely 1-byte per char.  The BSTR is 2 bytes per char.  At the point of returning the string, the memory usage will jump to 3 times the size of the JSON.  (If you have 250MB of JSON, returning the BSTR requires another 500MB — and it’s likely that the older programming language (FoxPro, VB6, etc.) cannot even handle a string that size, and there may be limitations within the COM infrastructure itself.

The solution is to NOT pass the JSON string back-and-forth.  This can be accomplished using the “Sb” methods — i.e. methods that pass the string in or out via a StringBuilder object.  This way, you’re just passing a reference to an object, and the string itself stays entirely within native (Chilkat) memory in utf-8 representation.   In this case, you would instead call EmitSb (instead of Emit) to write the JSON into a StringBuilder object, and then call StringBuilder.WriteFile.  However, this is still a doubling of memory usage because you’ll have a complete copy of the JSON within the JsonObject, and also within the StringBuilder (although it’s not as bad as the original case, and whatever limitations imposed by COM or the older programming language won’t exist).  But… if you’re only needing to get the JSON to save it to a file, the best solution is to call JsonObject.WriteFile to just write the JSON file directly using the JsonObject.