modern dyalog application development workflow?

[AntonioDiNarzo]

Hi all,

the short version of the post is as per subject line.

A more concrete rephrasing would be:

Can some nice soul help me setup a skeleton hello-world command line app with some basic unit/functional/whatever tests?

Verbose discussion below.

I'm trying to learn dyalog apl to see if it can help me in my data analysis tasks under linux. I went through few chapters of 'mastering dyalog apl' (a truly great book!). However, now I'm struggling since several days trying to go beyond simple interactive exercises in the [r]ide, to writing a simple application which loads some data, performs some computation, and spits the results back.

And this is where I'm terribly stumped and baffled, *way* more than learning to "think in arrays".

The cookbook seems to be oriented towards development of commercial GUI apps under windows. Nevertheless, I tried following the chapter on setting up a testing harness. That looked very opaque and verbose, both in coding the tests and producing the results, and requires massive imports from the 'APLTree/Testing' framework. Out of curiosity, I went checking on github what some other prominent dyalog apl projects use, and I saw 'APLTree/Testing' nowhere. I also found the 'aplunit' library, which seems stagnant without docs nor example projects I could learn from. "co-dfns" author seems to have developed his own testing system, same for "vecdb". Other dyalog projects on github seem to be using all different styles. The most familiar-looking approach to me seems to be what's used in e.g. conga, that is the dyalog internal 'dbuild'/'dtest' system. However documentation there is understandably lacking. Combined with my primitive knowledge of apl, setting up a basic template app with one failing and one passing test is turning out to be a major undertaking beyond my capabilities.
The fact that I cannot simply turn my dyalog code into #! scripts, means that I can't even resort to using external (non-apl) testing tools, and I'm stuck in the very ecosystem I'm trying to learn.

So, how do you guys do it? Am I looking at this all from the wrong angle? In my dyalog apl training, how would you suggest I transition from small interactive snippets in the ide, to writing bigger applications for my own personal use and learning?

thanks!

[paulmansour]

Hi Antonio,

A couple of questions for you first.

You mention trying to go beyond doing "simple interactive exercises in the ide."

Are you comfortable creating and calling functions, organizing them in namespaces, and saving your work? That is, do you have an application already written, that you can use interactively in the session (IDE)? And you just want to know how to call it from the command line?

Deployment of an app in a commercial setting is not trivial, one may need some additional tools for installers, help systems, etc., but writing for your own use, and setting up tests, should not require any heavy duty tools.

For example, a testing framework can simply be a namespace with a bunch of test functions, each of which returns 1 or 0. A simple cover function can run all these functions in one go for automated testing.

Be happy to expound more on this, but want to get a better picture of where exactly you are right now.

Regards,

Paul

[paulmansour]

Here is what I would do, somewhat guessing at your current circumstances.

In order to to this, you will need to be familiar with "workspaces", "namespaces" and "functions". You will need to know how to write and call functions, specifically how to call functions across namespaces. You will need to know how to create namespaces, move between them, and create functions in different namespaces.

Let's call the app you are going to write a "project". This is not a formal term in Dyalog, but it should, and I hope it becomes one soon.

In your workspace, create a top-level namespace to contain your project. This encapsulates your project so it can work with other projects - your own, or other peoples. The name of this namespace should be the name of the project.

In this project namespace, create 3 sub namespace named "Tests", "Admin" and "Main".
You will put your test functions in Tests, your application code in Main, and administrative code in Admin. For simple projects, these namespaces will have no sub namespaces, they will just contain functions. For complex applications, you would organize all your code in Main into various sub-namespaces to whatever level is appropriate.

In the Tests namespace, create a few test functions that simply return 1 or 0 - that do nothing.

In Admin, create a function that inspects the Tests namespace for test functions, and will run all of these test functions in Tests in one go. This is a trivial function. Name it "RunTests"

Now you have an awesome testing framework - no third party tools, a minimum amount of code, and if you are new to APL you will learn a lot of very useful things.

Now write a failing test, a test that calls some code in Main that does not exist.

Then go into Main and write the code until the test passes.

Repeat.

Not sure if this is what you are looking for, let me know, will be happy to help if I can.

Once you get your app doing a little bit, then it's time to discuss source code management and deployment.

Regards,

Paul

[AntonioDiNarzo]

Hi Paul.

Are you comfortable creating and calling functions, organizing them in namespaces, and saving your work? That is, do you have an application already written, that you can use interactively in the session (IDE)? And you just want to know how to call it from the command line?

I did create few functions which are saved in a namespace file.

As for running some code from the command line, I've got something working through wrappers around 'dyapp' files, though I do still wonder what is the recommended way to do it.

Deployment of an app in a commercial setting is not trivial, one may need some additional tools for installers, help systems, etc., but writing for your own use, and setting up tests, should not require any heavy duty tools.

I reckon that, and was indeed hoping for something lean.

For example, a testing framework can simply be a namespace with a bunch of test functions, each of which returns 1 or 0. A simple cover function can run all these functions in one go for automated testing.

That would be indeed the basic functionality I would expect from a testing framework, though I would have no idea on how to reliably (or unreliably, for what matters) implement such cover function myself.

Further, besides the basic '# of tests passing/failing/crashing', it would be nice to have some report of why something failed, say "expected 'hello', got 'world' instead". That seems to be the kind of stuff that APLTree gives me (?) but, as I said in my previous post, it seems fairly cumbersome to use, and from what I see on github, it doesn't exactly have a big following in the community, so much so that new projects simply don't use it.

[AntonioDiNarzo]

Here is what I would do, somewhat guessing at your current circumstances.

Sorry, I missed your intervening message. Thanks for your input, I'll go through it carefully.

create a function that inspects the Tests namespace for test functions, and will run all of these test functions in Tests in one go. This is a trivial function.

Hah, don't you tell me :-)

This means I should now how to list functions in one namespace from another one, and 'run' those function objects. I'm afraid for myself writing such a function would be a brand new 'project' in itself. I guess I'll put that into my 'learning apl' task backlog.

[paulmansour]

Antonio,

This is an interesting somewhat philosophical problem.

A good question to ask is, how much work is going to take me to use a third party testing framework? Or any framework for that matter? Then ask, is that work and effort transferable - that is, does it add to your permanent skill set?

Obviously this has to be weighed against the work it takes to "roll your own", and if that work and effort is transferable.

While I would never suggest you roll your own web browser, I think for the beginning APLer, with or without a lot of coding experience in other languages, writing you own test framework is not only worth the effort, but will in fact be much faster and less work than learning some 3rd party framework. In fact, it should probably be an exercise in an APL Introductory class.

What follows is a rudimentary test framework. I have written it as two functions, to make it easy to modify:

      RunTests←{
           t←##.Tests               ⍝ Tests namespace
           f←t.⎕NL ¯3               ⍝ List of the test functions
           b←RunTest¨f              ⍝ Run each test
           r←⊂'Tests run: ',⍕≢b     ⍝ Total run
           r,←⊂'Passed: ',⍕+/~b     ⍝ Total passed
           r,←⊂'Failed: ',⍕+/b      ⍝ Total failed
          ↑r
       }

      RunTest←{                        ⍝ Run a single test case function
           t←##.Tests                  ⍝ Test space
           b←t⍎⍵,' 0'                  ⍝ Execute the test function
           ⎕←⍵,': ',b⊃'Pass' 'Fail!'    ⍝ Display in session
           b                           ⍝ Result
        }

You can easily enhance this to add features you may want. As your app grows and matures, you will no doubt want to add some functionality.

Anything more than is, I would hazard a guess, is totally overkill for what you are doing.

The point is that knowing how to write, use, and enhance this little bit of code (framework is really too big a word) is a worthwhile basic skill.

Regards,

Paul

[AntonioDiNarzo]

thanks Paul. This snippet, together with the project layout, is of great help, and perhaps allows me to pull myself out of this rabbit hole.