Http Get from C# to APL

[paulmansour]

Morten,

We have considered adding a mechanism to Conga to declare that it should wait for a complete HTTP response before reporting anything back to APL

Now you tell me!!!! After I wrote all that code!

As you note, the client probably does not need to inspect the bytes flowing by, especially as the client can do a HEAD request if necessary. The only issue might be chunked responses. But even on the server, where it is critical that the header can be examined before reading the rest of the message, perhaps it would be useful to have Conga gather the entire message, but in two waits rather than one: The first could return the header, the second the body. This still gives the server a chance to reject the message, or send a 100-continue or what not. Again, an issue might be chunked messages from the client (not to common anyway).

I have no idea of the CPU time needed for the all of the accumulation and parsing of the HTTP message stream that we currently have to do in APL (am going to have some testing done in the coming months). For a busy server, do you think it would be big a improvement to have Conga do all this on a different thread/core whatever?

Paul

[Tomas Gustafsson]

For my part, it's a bit of a Google maps situation. But in a busy environment, since it happens during world render & logic. A data chunk comes in, or then it does not - no need to analyze it further during it's arrival. Timeout and cutoff options needed efc.

So why not a size limit? The TCP/IP MTU is a bit over 1400 bytes, and the single packets (in Google, as in any similar solution, that "wanders" over an indefinite amount of data) is probably (my guess!) a few 10's of kilobytes, perhaps in any case less that 100k? Multiple MTU's, maybe a few to 100. If Conga works on MTU units (which i guess it does), it could loop internally, if it concludes the expected data size is less than myLimit?

Modern 10mbit connections pass 1 Mbyte/second, so 100k should be a fraction of a second. Could really be completely externalized :-). And note, even a .net async style callback might be an option? Then APL wouldn't even need to hang on a suspended local thread? Instead, the callback would spawn the thread n, but it would exist only for the duration of receiving the complete result from Conga. 5 cents.

[Tomas Gustafsson]

We have considered adding a mechanism to Conga to declare that it should wait for a complete HTTP response before reporting anything back to APL

Sounds wise.

...you will need to do an "Export to Memory" - this should create a .net wrapper for that class. Alternatively you could pass some other reference to rs to BeginGetResponse, a unique name, or an index into an array for example.

On a related note, AsyncState generates a VALUE ERROR because the interpreter raises this error when a .Net method or property doesn't return a result (i.e. returns null). We are considering having the interpreter return ⎕NULL in this case, as this would be more symmetrical, and resolve some other inconsistencies in behaviour. We are interested in any and all feedback on this change.

Thanks John! I'll experiment.

⎕NULL might be more consistent, but would it be a bit more dangerous? A good halt is definite - could a ⎕NULL sneakily pass through? Hm... are there cases where i'd rather take a ⎕NULL for a missing object, than an error? I think my vote goes for VALUE ERROR, provied it appears only for the invalid part of the structure. Since the philosophy would be that it's better to avoid the problem entirely in advance, than reacting to it afterwards, when it appears. Ie. i'd fix it so that a VALUE ERROR will never ever appear :-).

However: A similar case, where i simply pass 0 as "Object", is below:

      #.SOCKET.hand←#.AsyncCallback.New ⎕OR'#._Socket.h_receiveasync'  ⍝ (Async) function for receiving incoming data
sync←#.SOCKET.socket.BeginReceive #.SOCKET.hand 0 ⍝ Async start of first receive (sync variable not needed, callback gets one as arg)

and:

      h_receiveasync arg;
 ep←#.ByRef.New⊂#.Net.IPEndPoint.New(#.Net.IPAddress.IPv6Any)0
 DATA←82 ⎕DR ¯128+256|128+#.SOCKET.socket.EndReceive arg ep

This works - arg isn't crucial here.

[Tomas Gustafsson]

I guess i could use a litte more help... John?

Look at this (note: the same ⎕USING as below is set for the _Http namespace):

      :Class RequestState

:Access Public

:using Dyalog,bridge132.dll
:using System,system.dll
:using System.Net
:using System.Threading
:using System.Text
:using System.IO

    :Field Public BUFFER_SIZE←1024
    :Field Public requestData←StringBuilder
    :Field Public BufferRead←Array ⍝ .CreateInstance Byte 10
    :Field Public request←HttpWebRequest
    :Field Public response←HttpWebResponse
    :Field Public streamResponse←Stream
    :Field Public StreamDecode←Encoding.UTF8.GetDecoder ⍝ Create Decoder for appropriate enconding type.
     
    ∇ make0
      :Access Public
      :Implements Constructor
     
      BufferRead←Array.CreateInstance Byte 1024 ⍝ Array.CreateInstance(#._Http.Type.GetType⊂'System.Byte')10
      requestData←StringBuilder.New String.Empty
      request←⎕NULL
      streamResponse←Stream.Null
      response←HttpWebResponse.New ⍬
    ∇

:EndClass

... then i create an instance of the class, and check rs.BufferRead:

      rs←⎕NEW _Http.RequestState ⋄ rs.BufferRead
System.Byte[]

It's correct, System.byte. Now i select "Export to memory" from the File menu. APL v14 doesn't show anything of that in the status window; it did in version 13, see below:

      Declared Assembly DyalogActiveWorkspace
Declared Module DyalogActiveWorkspace
	Declared Type _Http.RequestState
		Compiling Constructor "make0"
			Result type "<empty>" resolved to System.Void
		Compiled Constructor "make0"
			Field type "<empty>" resolved to System.Object
	Defined Field request
			Field type "<empty>" resolved to System.Object
	Defined Field requestData
			Field type "<empty>" resolved to System.Object
	Defined Field response
			Field type "<empty>" resolved to System.Object
	Defined Field streamResponse
			Field type "<empty>" resolved to System.Object
	Defined Field BufferRead
			Field type "<empty>" resolved to System.Object
	Defined Field BUFFER_SIZE
			Field type "<empty>" resolved to System.Object
	Defined Field StreamDecode
	Emitted Type _Http.RequestState
Emitted Assembly "DyalogActiveWorkspace"

Then i create rs again:

      )erase rs
      rs←⎕NEW _Http.RequestState ⋄ rs.BufferRead
0 0 0 0 0 0 0 0 0 0

Zeroes! Why is rs.BufferRead now zeroes, not System.Byte[], as it was before exporting to memory? Export to memorry makes .net recognize the rs object, but instead it has lost it's type definitions for the fields! Somehow the "exported to memory" version take dominance over the object creation, and makes a slightly different object.

The process then later on fails on rs.BufferRead - it never gets any byte content, the wrong object type seems to be dominating and blocking. For other parts, the async callbacks DO work; there is a ca 50-cycle loop), just that the retrieved data is missing. I'm sure it came to my computer (when totalling the 'len' values, i got 51892 bytes, very expected), just that rs.BufferRead cannot get it.

- What happens with the field type when exporting? Can i correct/fix it somehow, so that System.Byte[] would be persistent?

- In any case... i noticed that )save does NOT include the "export to memory" situation, meaning i'd have to export to memory each time i re-load the ws. I guess this is intentional?

- Is this really the situation, that whatever OO :Classes etc i define locally in APL code, are not valid for .net? I guess the APL OO is mostly intended for APL-only (non-,net interaction) coding, and then for exporting to assemblies? The APL OO is rather cool stuff though, so either i misunderstood it all (likely :-)), or there is a gap in the functionality?

[Tomas Gustafsson]

Bump

The object becomes different depending on execution of "Export to memory".

- Without a "Export to memory", .net/OS does not recognize the object at all => no go.

- With "Export to memory", the object loses/alters it's internal types, causing a mismatch/error in .net/OS, which does not populate an element of the object as it should => no go.

Esport to memory is not persistent, a ws )save does not include it; the "interpreter-local assembly" is no longer there upon next ws load. Can i export to memory programmatically (eg. as a proggy startup action)?

In general, i wondered whether this was intended to work in the first place (and i just haven't realised how), or is this inherently a no go? In C#, they create objects containing .net types and pass to .net, and since i can stylishly do the same creation in APL, it led me to believe it might be possible to do the full implementation.

These are questions i really have no way to resolve myself, so it would be most useful to get a bit of input :-).

[JohnD|Dyalog]

Hi Thomas,

This is interesting.

In almost all cases when the interpreter gets a .NET array from the .NET interface it is converted to an APL array and so it becomes an array like any other in the workspace. However, an exception to this rule is when you call Array.CreateInstance. In this case we create an object in the workspace that encapsulates the .NET array. As you've seen, the "display form" of this object is different. Clearly there would be no point in calling Array.CreateInstance if we just returned an APL array, and similarly you wouldn't want all arrays returned from .NET to be .NET objects rather than APL arrays.

Now, when you do the export to memory the interpreter creates the .NET class definition and .NET metadata for the object. Subsequently all queries of fields and properties (and calls to methods) go through the .NET interface to get the value so the array. This means that this time the .NET array is converted to an APL array, and you get the content of the array not a wrapper around it.

The behaviour you see then, is an unintended side-effect of the special casing of Array.CreateInstance.

I think you will find that once the response is received and the array has been populated rs.BufferRead will return the correct data.

Best Regards,
John Daintree.