Sharing data between APL sessons

[ray]

Hi,

I have a classic workspace updating (in a loop) some data in memory in real time. (The “DATA”)

I also have a Class being used by MiServer.

I would like the Class to have be able to get snapshots of the “DATA” and present that to MiServer without embedding code from the classic workspace into the Class.

The amount of data being transferred at each loop is not large, normally much less than a megabyte.

I can think of several methods:

1) Component file.
At the end of each “loop” a copy of the DATA is appended as a new component.
The MiServer class reads each component in turn and displays it.

2) Use Shared Variables (⎕SVC ⎕SVO ⎕SVQ ⎕SVR and ⎕SVS)
It a long time since I tried this, but it should be faster than 1)

3) Making the workspace an OLE server and the Class an OLE client
2) uses DDE but 3) uses COM so should be preferred. I however have no experience of using APL at both ends of an OLE link.

4) Using TCP/IP Sockets
Is this restricted to running under Windows, or can it be run under Linux

5) Using ⎕SAVE and ⎕CY

What method would you suggest and why?

[Dick Bowman]

With my belief in the simple life I'd go with the component file option - unless performance really was critical.

Rejecting Shared Variables and OLE because I think (probably wrongly) that they are old hat.

Rejecting ⎕SAVE/⎕CY because I've done similar in the past and found it clunky (also sounds a bit too APLy).

If I knew more about techy stuff I'd feel TCP/IP sockets were probably the way to go (but also feel a bit wary about performance there).

[AndyS|Dyalog]

My £0.02: I would go the socket route.

1. Using sockets gives you the most flexibility in platforms that you can run on, and allows you to separate the data generater and the MiServer onto two different machines. Using sockets also doesn't impose any limits on the platform that you must run on; both TryAPL and the Dyalog '14 registation system were developed on Windows, but run/ran in production on small Linux VMs.

2. If you go the socket route, use Conga rather than native TCPSockets .. both work on all platforms supported by Dyalog, but Conga is the way to go - and is used by MiServer and DFS. DFS is all about passing APL arrays between Dyalog processes.

If I was to not use sockets, I'd probably go for the component file route - as Dick says, it's simple, and is platform independent.

If you're using 14.0 then you can use 219⌶ and 220⌶ or compressed components to minimise the amount of data you pass between the processes. See http://help.dyalog.com/14.0/index_CSH.htm#Language/Primitive%20Operators/I%20Beam.htm and http://help.dyalog.com/14.0/index_CSH.htm#RelNotes14.0/Compressed%20Components.htm for more information.

[ray]

Thanks Dick, at the moment that is also my "KISS" (keep it simple) option.

One other option that I might consider is to use eXTernal variables. However I am currently heeding the warning Dyalog keeps making at the "conf-user meetings" so have excluded it for the time being.

One advantage of the component file system method, it would allow me to run the MiServer class on the file produced by the classic workspace, long after the workspace has completed. If the component contains both "before and after" data rather than just "after" data, then it would also allow me to "run" it forwards and backwards. None of the other options allow for "running" it backwards.

[ray]

Thanks Andy.

Conga will allow the MiServer and my Classic Workspace to be on different devices, and is not limited to just Windows.

In the mean time, Gil has suggested (via Skype) using memory mapped files, which, so long as I can fix the data format, will give me the speed, albeit forcing the MiServer and Workspace to be on the same device.

To see the current view of my MiServer (currently not accessing the data from my Classic Workspace) go to
86.27.228.220:8080/hex
(This will not be up 24/7 as my PC goes to sleep and does not wake-up-on-lan.)

[Morten|Dyalog]

One advantage of the component file system method, it would allow me to run the MiServer class on the file produced by the classic workspace, long after the workspace has completed.

You can supplement this with the use of Conga for notification when the data has changed - you could use a normal HTTP get message to a pre-agreed URL to send the "message" to the MiServer.

[ray]

After trying memory mapped files, and failed to get the MiServer class reading the file as the Classic Workspace updated (wrote) it, I converted to using a simple component file, and it is now working very nicely.

If my MiServer class is running, you should be able to see it in action at
//86.27.228.220:8080/hex

I now need to tidy it up and "refactor" it. There are some horrible loops, and ⎕FMT statements used to create my dynamic JSCRIPTs. Also the colours leave a lot to be desired.

I will be trying a Conga version to see how much faster it might be.

Thank you all for all your help and suggestions, especially Brian for his help with MiServer 3.0 at Eastbourne