Question about the new ⎕JSON coming up with v16
5 posts
• Page 1 of 1
Question about the new ⎕JSON coming up with v16
We are using the Dyalog JSON script to serialize/de-serialize Apl data to text file like this: [2,2,3,1,2,3,4,5,6] ←→ 2 3⍴⍳6 . The script is rather slow, will the new ⎕JSON will be able to do what the Dyalog JSON script is doing. In the negative why not ? Is there another way to serialize/de-serialize apl data to text file that is quick and supported by Dyalog ?
Thanks in advance,
Pierre Gilbert
Thanks in advance,
Pierre Gilbert
-
PGilbert - Posts: 436
- Joined: Sun Dec 13, 2009 8:46 pm
- Location: Montréal, Québec, Canada
Re: Question about the new ⎕JSON coming up with v16
Hey Pierre
This will do the job for simple data or down to depth two:
There are other problems with 7160⌶ concerned with empties. It encodes both ⍬ and 0⍴⊂⍬ as the same empty list [] This doesn't overcome that.
Controlling when to use recursion and decode are left to the reader.
This will do the job for simple data or down to depth two:
encode←7160⌶∘{(⍴⍴⍵)(⍴⍵)(,⍵)}- substitute ⎕JSON for 7160⌶ where available - but requires a doubly recursive call for stuff more deeply nested
encode{⍺⍺ ⍺⍺¨⍵}There will certainly be other edge cases that cause problems but this'll do most of what you want.
There are other problems with 7160⌶ concerned with empties. It encodes both ⍬ and 0⍴⊂⍬ as the same empty list [] This doesn't overcome that.
Controlling when to use recursion and decode are left to the reader.
-
Phil Last - Posts: 628
- Joined: Thu Jun 18, 2009 6:29 pm
- Location: Wessex
Re: Question about the new ⎕JSON coming up with v16
Of course you will have realised that as the control data are prefixed without catenation we don't actually need to include the rank.
encode←7160⌶∘{(⍴⍵)(,⍵)} ⋄ decode←{⊃⍴/7159⌶⍵}
-
Phil Last - Posts: 628
- Joined: Thu Jun 18, 2009 6:29 pm
- Location: Wessex
Re: Question about the new ⎕JSON coming up with v16
v16 will include ⎕JSON as a replacement for the experimental I-Beams (which will be retained for one release in order to allow for an orderly transition). The option to add metadata and convert APL arrays into something that can be represented in JSON was provided by the JSON utility library, but is not included in the new definition of ⎕JSON (nor was it supported by the I-Beams).
In an e-mail, you sent me some data to play with: The only problem you have is that some of your variables are matrices: JSON only supports vectors. I experimented with the following alternatives for serialisation, which are sufficient if you only have ranks greater than 1 at the top level.
As we have already seen, one strategy is to add shape information to each array and ravel the data:
Second alternative: Add an extra variable called METADATA, containing the shapes (and ravel the data):
The performance of the alternatives is almost identical. On my laptop, the old JSON namespace functions use 180ms + 230ms to convert your from APL to JSON and back (the JSON representation uses about 130kb to represent your 33 variables). The new functions use 15ms + 5ms; just over 10x faster for the serialisation and nearly 50x faster when reconstructing the namespace. The extra lines of code don't seem to add much to the time spend by the system function.
A final point worth making: If you want to serialise data as JSON, you might want to consider using data structures which are all directly supported (in other words, no matrices). Then you will just be able to use ⎕JSON directly, with the added benefit that non-APL code will be able to understand the data too, without additional special knowledge.
In an e-mail, you sent me some data to play with: The only problem you have is that some of your variables are matrices: JSON only supports vectors. I experimented with the following alternatives for serialisation, which are sufficient if you only have ranks greater than 1 at the top level.
As we have already seen, one strategy is to add shape information to each array and ravel the data:
- Code: Select all
∇ r←toJSON2 ns;vars;ns2
[1] 'ns2'⎕NS ⎕OR'ns' ⍝ clone
[2] vars←⍕ns2.⎕NL-2
[3] ns2⍎'(',vars,')←{(⍴⍵) (,⍵)}¨',vars
[4] r←⎕JSON ns2
∇
∇ r←fromJSON2 json;vars
[1] r←⎕JSON json
[2] vars←⍕r.⎕NL-2
[3] r⍎'(',vars,')←⊃¨⍴/¨',vars
∇
Second alternative: Add an extra variable called METADATA, containing the shapes (and ravel the data):
- Code: Select all
∇ r←toJSON3 ns;vars;ns2
[1] 'ns2'⎕NS ⎕OR'ns' ⍝ clone
[2] ns2.(METADATA←{⍵(⍴¨⍎⍕⍵)}⎕NL ¯2)
[3] vars←⍕1⊃ns2.METADATA
[4] ns2⍎vars,'←,¨',vars
[5] r←⎕JSON ns2
∇
∇ r←fromJSON3 json;vars;shapes
[1] r←⎕JSON json
[2] vars←⍕1⊃r.METADATA
[3] r⍎'(',vars,')←(2⊃METADATA)⍴¨',vars
[4] r.⎕EX'METADATA'
∇
The performance of the alternatives is almost identical. On my laptop, the old JSON namespace functions use 180ms + 230ms to convert your from APL to JSON and back (the JSON representation uses about 130kb to represent your 33 variables). The new functions use 15ms + 5ms; just over 10x faster for the serialisation and nearly 50x faster when reconstructing the namespace. The extra lines of code don't seem to add much to the time spend by the system function.
A final point worth making: If you want to serialise data as JSON, you might want to consider using data structures which are all directly supported (in other words, no matrices). Then you will just be able to use ⎕JSON directly, with the added benefit that non-APL code will be able to understand the data too, without additional special knowledge.
-
Morten|Dyalog - Posts: 453
- Joined: Tue Sep 09, 2008 3:52 pm
Re: Question about the new ⎕JSON coming up with v16
Thanks Morten and Phil, this is answering my question and Wow the code is extremely fast! For the benefit of everybody here is the functions that I was looking for:
- Code: Select all
json←SimpleAplToJSON apl;vars;apl2
⍝ Converts Simple APL Variable and Nameless Namespace of Simple APL Variables to JSON
⍝ Works with the function 'JSONtoSimpleApl' to Deserialize the JSON back to APL data
⍝ Code taken from: http://www.dyalog.com/forum/viewtopic.php?f=13&t=1118
⍝
⍝ Examples:
⍝ ⍳5 ←→ [[5],[1,2,3,4,5]]
⍝ 2 5⍴⍳10 ←→ [[2,5],[1,2,3,4,5,6,7,8,9,10]]
⍝
⍝ ns ← ⎕ns''
⍝ ns.data1 ← ⍳5
⍝ ns.data2 ← 2 5⍴⍳10
⍝ ns.data3 ← 'This is some text'
⍝ ns ←→ {"data1":[[5],[1,2,3,4,5]],"data2":[[2,5],[1,2,3,4,5,6,7,8,9,10]],"data3":[[17],"This is some text"]}
:If 2=⎕NC'apl'
⍝ Simple APL variable
json←7160⌶{(⍴⍵)(,⍵)}apl
:ElseIf 9=⎕NC'apl'
⍝ Nameless Namespace of Simple APL Variables
'apl2'⎕NS ⎕OR'apl' ⍝ clone
vars←⍕apl2.⎕NL-2
apl2⍎'(',vars,')←{(⍴⍵) (,⍵)}¨',vars
json←7160⌶apl2
:Else
⍝ Don't know what to do
json←⎕NULL
:EndIf
- Code: Select all
apl←JSONtoSimpleApl json;vars
⍝ Converts JSON to Simple APL Variable or Nameless Namespace of Simple APL Variables
⍝ Works with the function 'SimpleAplToJSON' to Serialize the APL data to JSON
⍝ Code taken from: http://www.dyalog.com/forum/viewtopic.php?f=13&t=1118
⍝ See 'SimpleAplToJSON' for examples
:If '['=1↑json
⍝ Simple APL variable
apl←⊃⍴/7159⌶json
:ElseIf '{'=1↑json
⍝ Nameless Namespace of Simple APL Variables
apl←7159⌶json
vars←⍕apl.⎕NL-2
apl⍎'(',vars,')←⊃¨⍴/¨',vars
:Else
apl←⎕NULL
:EndIf
-
PGilbert - Posts: 436
- Joined: Sun Dec 13, 2009 8:46 pm
- Location: Montréal, Québec, Canada
5 posts
• Page 1 of 1
Who is online
Users browsing this forum: No registered users and 1 guest
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group