Question about the new ⎕JSON coming up with v16

[PGilbert]

We are using the Dyalog JSON script to serialize/de-serialize Apl data to text file like this: [2,2,3,1,2,3,4,5,6] ←→ 2 3⍴⍳6 . The script is rather slow, will the new ⎕JSON will be able to do what the Dyalog JSON script is doing. In the negative why not ? Is there another way to serialize/de-serialize apl data to text file that is quick and supported by Dyalog ?

Thanks in advance,

Pierre Gilbert

[Phil Last]

Hey Pierre
This will do the job for simple data or down to depth two:

      encode←7160⌶∘{(⍴⍴⍵)(⍴⍵)(,⍵)}

- substitute ⎕JSON for 7160⌶ where available - but requires a doubly recursive call for stuff more deeply nested

      encode{⍺⍺ ⍺⍺¨⍵}

There will certainly be other edge cases that cause problems but this'll do most of what you want.
There are other problems with 7160⌶ concerned with empties. It encodes both ⍬ and 0⍴⊂⍬ as the same empty list [] This doesn't overcome that.
Controlling when to use recursion and decode are left to the reader.

[Phil Last]

Of course you will have realised that as the control data are prefixed without catenation we don't actually need to include the rank.

      encode←7160⌶∘{(⍴⍵)(,⍵)} ⋄ decode←{⊃⍴/7159⌶⍵}

[Morten|Dyalog]

v16 will include ⎕JSON as a replacement for the experimental I-Beams (which will be retained for one release in order to allow for an orderly transition). The option to add metadata and convert APL arrays into something that can be represented in JSON was provided by the JSON utility library, but is not included in the new definition of ⎕JSON (nor was it supported by the I-Beams).

In an e-mail, you sent me some data to play with: The only problem you have is that some of your variables are matrices: JSON only supports vectors. I experimented with the following alternatives for serialisation, which are sufficient if you only have ranks greater than 1 at the top level.

As we have already seen, one strategy is to add shape information to each array and ravel the data:

Code: Select all

     ∇ r←toJSON2 ns;vars;ns2
[1]    'ns2'⎕NS ⎕OR'ns' ⍝ clone
[2]    vars←⍕ns2.⎕NL-2
[3]    ns2⍎'(',vars,')←{(⍴⍵) (,⍵)}¨',vars
[4]    r←⎕JSON ns2
     ∇
     ∇ r←fromJSON2 json;vars
[1]    r←⎕JSON json
[2]    vars←⍕r.⎕NL-2
[3]    r⍎'(',vars,')←⊃¨⍴/¨',vars
     ∇


Second alternative: Add an extra variable called METADATA, containing the shapes (and ravel the data):

Code: Select all

     ∇ r←toJSON3 ns;vars;ns2
[1]    'ns2'⎕NS ⎕OR'ns' ⍝ clone
[2]    ns2.(METADATA←{⍵(⍴¨⍎⍕⍵)}⎕NL ¯2)
[3]    vars←⍕1⊃ns2.METADATA
[4]    ns2⍎vars,'←,¨',vars
[5]    r←⎕JSON ns2
     ∇
     ∇ r←fromJSON3 json;vars;shapes
[1]    r←⎕JSON json
[2]    vars←⍕1⊃r.METADATA
[3]    r⍎'(',vars,')←(2⊃METADATA)⍴¨',vars
[4]    r.⎕EX'METADATA'
     ∇

The performance of the alternatives is almost identical. On my laptop, the old JSON namespace functions use 180ms + 230ms to convert your from APL to JSON and back (the JSON representation uses about 130kb to represent your 33 variables). The new functions use 15ms + 5ms; just over 10x faster for the serialisation and nearly 50x faster when reconstructing the namespace. The extra lines of code don't seem to add much to the time spend by the system function.

A final point worth making: If you want to serialise data as JSON, you might want to consider using data structures which are all directly supported (in other words, no matrices). Then you will just be able to use ⎕JSON directly, with the added benefit that non-APL code will be able to understand the data too, without additional special knowledge.

[PGilbert]

Thanks Morten and Phil, this is answering my question and Wow the code is extremely fast! For the benefit of everybody here is the functions that I was looking for:

Code: Select all

 json←SimpleAplToJSON apl;vars;apl2
⍝ Converts Simple APL Variable and Nameless Namespace of Simple APL Variables to JSON
⍝ Works with the function 'JSONtoSimpleApl' to Deserialize the JSON back to APL data
⍝ Code taken from: http://www.dyalog.com/forum/viewtopic.php?f=13&t=1118
⍝
⍝ Examples:
⍝ ⍳5      ←→ [[5],[1,2,3,4,5]]
⍝ 2 5⍴⍳10 ←→ [[2,5],[1,2,3,4,5,6,7,8,9,10]]
⍝
⍝ ns ← ⎕ns''
⍝ ns.data1 ← ⍳5
⍝ ns.data2 ← 2 5⍴⍳10
⍝ ns.data3 ← 'This is some text'
⍝ ns ←→ {"data1":[[5],[1,2,3,4,5]],"data2":[[2,5],[1,2,3,4,5,6,7,8,9,10]],"data3":[[17],"This is some text"]}

 :If 2=⎕NC'apl'
    ⍝ Simple APL variable
     json←7160⌶{(⍴⍵)(,⍵)}apl

 :ElseIf 9=⎕NC'apl'
    ⍝ Nameless Namespace of Simple APL Variables
     'apl2'⎕NS ⎕OR'apl' ⍝ clone
     vars←⍕apl2.⎕NL-2
     apl2⍎'(',vars,')←{(⍴⍵) (,⍵)}¨',vars
     json←7160⌶apl2

 :Else
    ⍝ Don't know what to do
     json←⎕NULL

 :EndIf


Code: Select all

 apl←JSONtoSimpleApl json;vars
⍝ Converts JSON to Simple APL Variable or Nameless Namespace of Simple APL Variables
⍝ Works with the function 'SimpleAplToJSON' to Serialize the APL data to JSON
⍝ Code taken from: http://www.dyalog.com/forum/viewtopic.php?f=13&t=1118
⍝ See 'SimpleAplToJSON' for examples

 :If '['=1↑json
    ⍝ Simple APL variable
     apl←⊃⍴/7159⌶json

 :ElseIf '{'=1↑json
    ⍝ Nameless Namespace of Simple APL Variables
     apl←7159⌶json
     vars←⍕apl.⎕NL-2
     apl⍎'(',vars,')←⊃¨⍴/¨',vars

 :Else
     apl←⎕NULL

 :EndIf