Enhanced Debugging with Function Keys – Evaluate selection

See also Enhanced Debugging with Function Keys.

When tracing through a complex dfn and reaching a guard (condition:result), I am often wary of tracing into that line because if the condition evaluates to 1 then the current function I’m tracing through will terminate and return the result, leading to me losing situational awareness. Normally, I’d select the condition expression, copy it, move to the session and execute the expression, so I can predict what will happen next. Can we automate this? Yes we can.

Now, I usually prefer the Windows IDE for my daily development, but this is actually a case where RIDE has neat feature that’s missing from the IDE (but if you keep reading, I’ll show you how to achieve a similar effect in the IDE). In RIDE, go to Edit ⇒ Preferences ⇒ Shortcuts (or simply click ⌨︎ at the right end of the language bar), then type the name of a function key you want to use for this purpose, followed by a space, for example “F6 ” for . You’ll see exactly one entry in the listing. In the input field, write “<VAL>” (without quotes):

I defined a simple function to test it with, and traced into that:

      ⎕VR⎕FX'f←{' '⍺∧⍵:''both''' '⍺∨⍵:''either''' '''neither''' '}'
     ∇ f←{
[1]        ⍺∧⍵:'both'
[2]        ⍺∨⍵:'either'
[3]        'neither'
[4]    }
     ∇ 
      f

Tracing into f
Upon reaching a guard, I select the condition:
Selecting the condition
And Press :
Pressing F6
Voilà!

Cool, but how about the IDE?

Right, the Windows IDE doesn’t support the VAL command code, but we can easily emulate it by combining multiple command codes and assigning them to an F-key using ⎕PFKEY.

What we need to do is:

  1. Copy the current selection
  2. Jump to the session
  3. Paste
  4. Execute
  5. Jump back again

Options ⇒ Configure… ⇒ Keyboard Shortcuts ⇒ Description gives that the command codes for “Copy”, “JumP between current window and session window”, and “Paste” are CP, JP, and PT. We use ER (you can find all but JP using the ]KeyPress user command too) to press . Here we go:

      'CP' 'JP' 'PT' 'ER' 'JP' ⎕PFKEY 6
┌──┬──┬──┬──┬──┐
│CP│JP│PT│ER│JP│
└──┴──┴──┴──┴──┘

Keep it so!

RIDE keeps its setting, but of course, I wouldn’t want to be bothered with setting this up for every IDE session. So here’s a trick to set up F-keys (or anything else for that matter). When Dyalog APL starts up, it will look for MyUCMDs\setup.dyalog in your Documents folder ($HOME/MyUCMDs/setup.dyalog on non-Windows). If this file contains a function named Setup, it will be run whenever APL starts:

      ∇Setup
[1]  '<F6> is: ','CP' 'JP' 'PT' 'ER' 'JP' ⎕PFKEY 6
[2]  ∇
      (⊂⎕NR'Setup')⎕NPUT'C:\Users\Adam.DYALOG\Documents\MyUCMDs\setup.dyalog'

And now, when I start APL:
Upon start

Dyalog ’18 Videos, Week 6

Happy New Year – and Welcome to the 6th week of Dyalog ’18 video releases!

If you enjoy geometry, 2019 starts with a couple of real treats; one which builds up to the use of complex numbers just before the end, and another which starts with them and moves on to Quaternions. Alternatively, if you think vectors and matrices containing imaginary numbers are a bit esoteric, what could be more “down to earth” than taking a look at various ways to efficiently extract data from Excel spreadsheets? Finally, we have a talk on a Theory of Everything, which will obviously interest everyone!

Returning to the maths: Nic Delcros asks a seemingly trivial question about the number of dimensions of a vector. As any APLer knows, a vector is a list of numbers and, therefore, has 1 dimension, but of course the numbers in a vector nearly always represent a structure of higher dimensionality. Nic takes us on an entertaining exploration of the case where the numbers represent a dynamic event, where one of the dimensions is time – punctuated with beautiful images.

Dieter Kilsch from the University of Applied Sciences (Technische Hochschule) in Bingen obviously enjoys teaching mathematics! In this talk, he actually managed to make me think that I had some insight into why the Irish mathematician William Hamilton invented the Hamiltonian number system (which is populated by Quaternions), and how it allows us to do algebra on points in a 3-dimensional space, similar to the way complex numbers work for 2 dimensions. For example, Quaternions can be used as a tool of thought and computation for image recognition!

Returning to the very real world, Richard Procter is back with an updated talk on “Excel Mining”, following on from his talk at Dyalog ’15 in Sicily. Like many of us, he frequently needs to load data which originates in Microsoft Excel into APL for processing – and sometimes write back to Excel. Richard has tried a variety of different techniques and provides a list of questions that might decide which technique to use in a given scenario (and performance measurements as well).

It should be no big surprise that John Daintree’s big TOE is not something he needs to take a shoe off to demonstrate. Rather, the Theory Of Everything is a unifying idea that might one day replace a large number of system functions, “root methods” and I-Beams which currently allow programmers to ask questions about the Universe that they are running in. The result will hopefully be a system that is more powerful, but simpler and much more self-documenting than the collection of tools that it would replace.

Summary of this week’s videos:

 

Dyalog ’18 Videos, Week 4

This week is mostly a deep dive into the new world of storing source code in text files rather than workspaces and other “binary” formats. However, in case that is not your cup of tea yet, we can offer you another talk by Marshall Lochbaum, who presents more amazing algorithms to make the very widely used primitive search functions ∊, ⍳ and ⍸ run faster than ever before. By combining non-branching algorithms with vector instructions and a technique known as Robin Hood Hashing, Marshall is able to drive a modern CPU close to the theoretical maximum throughput, and in many cases spend less than one nanosecond searching for each item of an array.

Source code in text files is the dominant theme, and we are fortunate enough to have three pioneers to show us the way: Paul Mansour, Gilgamesh Athoraya and Kai Jaeger.

Paul has been working on – and using – source code management systems for decades. Recently, his team have implemented a lightweight version of the Acre project management system, named Acre Desktop, based entirely on textual source files. Apart from having to start your day by ]Open-ing a project, rather than by )LOAD-ing a workspace, there are very few changes to how you would actually use APL – but now you have access to a huge collection of professional tools developed for programmers using other programming languages, such as GitHub.

One of the very significant advantages of the APL community starting to use common structures for source code – and projects – is that it becomes realistic to share tools and utilities.
Following on from Paul’s talk, Gilgamesh Athoraya demonstrates a prototype of an APL Package Manager (APM). The APM connects to a repository of packages written in APL and allows you to declare package depenedencies from a public or private repository. It also keeps tabs on the availability of new versions of dependencies, and allows you to easily update them when the time is right.

A package manager can only be successful if there are packages to be managed. Kai Jaeger has been an APL Toolsmith for a very long time, and made much of his work available via the APLTree. Now, Kai has transferred the contents of the APLTree to GitHub, making everything available as textual source. With a bit of luck, once the APM finds its legs, we’ll all be able to use Acre Desktop to define projects, Git[Hub/Lab] to manage the source, and APM to search for Kai’s tools and manage our dependencies on them!

Summary of this week’s videos:

Dyalog ’18 Videos, Week 3

The four presentations from Dyalog’18 that we are releasing this week address both the visible (user interface) and invisible (performance) parts of application design. Starting with performance:

“You don’t have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy.” – former Formula One racing driver Sir John Young “Jackie” Stewart, OBE


This quote was at the heart of the talk by our invited keynote speaker Martin Thompson. In order to write software which performs well, you need to have a basic understanding of how the underlying machinery works. Understanding basic mathematical models for the theoretical throughput of software and hardware helps us take the step from being alchemists to scientists, as we endeavour to write high-performance systems.

Martin takes us for an entertaining stroll through the evolution of modern processors, and some of the maths behind high performance systems. The good news is that systems which make sequential and predictable memory accesses are likely to find sympathy with modern hardware…

Marshall Lochbaum, the most recent addition to the core interpreter team at Dyalog, followed up with a talk on a number of his ideas for increasing the mechanical sympathy of Dyalog APL, to take maximum advantage of branch prediction and other features of modern processors. Some strategies take advantage of runtime inspection of the arguments, something that is more natural in an interpreter with the ability to dynamically select data types, as opposed to strongly typed strategies typically employed by compilers.


TamStat is an application which helps students Tame Statistics. In two talks at Dyalog’18, Stephen Mansour and Michael Baas focus on two different aspects of the user experience. In the first talk, Stephen focuses on the notation available to users of TamStat. Where many statistical libraries contain dozens of strangely named functions with a variety of switches and parameters, TamStat uses a small set of functions, combined with another small set of operators, to provide a very simple but extremely elegant notation for computing probabilities based on a wide variety of distributions. For example:

⍝ Probability that 7 coin flips (0.5 specifying a "fair" coin) will result 
⍝ in at least 3 heads:
7 0.5 binomial probability ≥ 3
⍝ Probability that a number from a normal distribution with a mean of 0 and 
⍝ standard deviation of 1 will be ≤ 3:
0 1   normal   probability ≤ 3

I almost wish I could go back to University and start Statistics 101 again 😊.


Notation is a powerful tool of thought, but graphs make it easier to visualise the results. Following Stephen’s talk, Michael Baas describes work that Dyalog is doing in collaboration with Stephen, with the goal of wrapping TamStat in a modern, HTML/JavaScript based frontend. Current TamStat is based on the ⎕WC (Window Create) library function and is therefore restricted to running on Microsoft Windows. However, many of Stephen’s students use Mac or Linux laptops. The new interface also makes it possible to run TamStat as a web-based service with a web site. We expect that this work will make TamStat accessible to a much wider audience.

Summary of this week’s videos:

Displaying cross-references with SharpPlot

Requirements: Update SharpPlot to v3.37 from my.dyalog.com > Downloads > Tools & Interfaces > GUI Tools > SharpPlot

SharpPlot v3.37 introduces Network Maps as a new chart type, and we’re going to use it to display the output of the ]XRef user command, which displays cross-references between all kinds of APL names :

      ]XRef ⎕SE.Parser
             DDEEEFLLMMNNPPPPPPPPPPPQQSSSTUaaaabbbcdddddddfffiilmmmmmmnnnnnnppppqqqqrrrsssssssssstttttuvvvxx∆⍵⍺ 
             EeRnrOOSAOAsRrsssssssss.uwwwrPrrrr:alu.aaaeee.ei.fq.aeioo.adeop.Daa.1mq.et:eipqtwwww:aqwxp.an.C... 
             LlRd:RW.XDRwEo:.eeeeeee.oDTiaPgggg:d.t.tttQff.ax.:..smndd..awv..art..:..qb:t.lzr.imp:b.vtp.lc.u... 
             I.OT:CE.APGiFp:Pttttttt.t.atpE.psv:....aaau.i.tC.:..k.lei..:....tm...:....:..i.:.tao:l...e....t... 
             M.Rr:ER.ROStIa:a........e.bc.R.o.a:....:..o.n.ua.:....elf..:....as...:....:..t.:..ts:e...r....:... 
             I.0a:S..GS.cXg:r.Samnpu.s.lh.:.s.l:....:ASt.e.rs.:....n.i..:....:....:....:..P.:....:....C....:... 
             T..p:P..S:.h.a:s.wloarp.:.e..:...u:....:rwe.d.ee.:....:.e..:....:....:....:..a.:....:....a....:... 
             E...:A...:.e.t:e.ildrep.:....:...e:....:gD..:.s..:....:.r..:....:....:....:..r.:....:....s....:... 
             R...:C...:.s.e:..toigfe.:....:...s:....:u...:....:....:.s..:....:....:....:..m.:....:....e....:... 
             ....:E...:....:..cwfsir.:....:....:....:m...:....:....:....:....:....:....:..s.:....:....:....:... 
             ....:....:....:..hni.x..:....:....:....:e...:....:....:....:....:....:....:....:....:....:....:... 
             ....:....:....:...oe....:....:....:....:n...:....:....:....:....:....:....:....:....:....:....:... 
             ....:....:....:...sr....:....:....:....:t...:....:....:....:....:....:....:....:....:....:....:... 
             ....:....:....:...ps....:....:....:....:s...:....:....:....:....:....:....:....:....:....:....:... 
             ....:....:....:...a:....:....:....:....:....:....:....:....:....:....:....:....:....:....:....:... 
             ....:....:....:...c:....:....:....:....:....:....:....:....:....:....:....:....:....:....:....:... 
             ....:....:....:...e:....:....:....:....:....:....:....:....:....:....:....:....:....:....:....:... 
 [FNS]        - - - - : - - - - : - - - - : - - - - : - - - - : - - - - : - - - - : - - - - : - - - - : - - - - 
 Parse       G.GG○G G GGGG. . . : . ○F.G.G:○○○○○! . ○GGF.○. .○F ○ .○. . ○!○○○G○○! : .○○○○ F :○○○○○○ ○ : ○○.F! . 
 Propagate    ○ . . . : . . . . : . . G . : . .○. . : . . . . : . . .○. : . . . . : . . .○. ○○. . . . :○. . . . 
 Quotes       . . . . : . . . . : . ○ . . : . . . . : . . . .○: .○. . ○ : . . . . ○ . . . . ○ . . .○. : . . . . 
 Switch       . G . . : . . . . : . . G .G: . . . . : . ○ . . : . . . . : . . . . : ○ .○. . : . . . . :○. . . . 
 deQuote      . . . . : . . . . : . . . . : . . . . : . . . . :○○ . . . : . . . . : . . . . ○ . . . . : . . . . 
 fixCase      . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . .○. 
 if           . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . 
 init        G.G. G GGGGGGFGGGGGGGGG.F.GF :○. .○. ○ : . . ○○F F . ○ ○ ○○: . .G. . : . . . .F: . . . . F . GF. . 
 splitParms   . . . . : . . . . : . ○ . . : . . .○.○: . . . . : . . . ○ : .○○ ○ ○○:○. .○. . : . .○. .○: . . . . 
 sqz          . . . . : . . . . : . . . . : . .○. . : . . . . : . . . . : . . . . : . .○. . : . . . . : . . .○. 
 upperCase    . . .G. : . . . . : . . . . G . .○. . : . . . .○: . . . . : . . . . : . .○. . : . . . . : . . .○. 
 xCut         . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . . . : . . .○○ 

We can capture the output of the user command by doing either
]mat←XRef ⎕SE.Parser (from session)
or
mat←⎕SE.UCMD']XRef ⎕SE.Parser' (from code)

Until Dyalog v16.0 (which has ]XRef -raw), we need to parse the output of ]XRef to get a square link matrix, along with the full list of nodes (in row/column order of the matrix) and list of original rows (functions).

    ∇ (mat nodes rows)←GetMatrix mat;nodelabs;rows;cols;labs;cat;diag
      (rows mat)←2↑(1,2</∧⌿mat=' ')⊂[2]mat        ⍝ cut at empty column
      (cols mat)←2↑(1,2<⌿∧/mat∊' -:')⊂[1]mat      ⍝ cut at - - - - : - - - -
      mat←1↓[1](' '∨.≠mat)/mat                    ⍝ trim first row and empty columns
      rows←~∘' '¨↓(-1⊃⍴mat)↑[1]rows               ⍝ row titles (list of strings)
      cols←~∘' .:'¨↓(-2⊃⍴mat)↑[1]⍉cols            ⍝ col titles (list of strings)
      nodes←rows∪cols                             ⍝ all titles
      mat←((mat,' ')⍪' ')[rows⍳nodes;cols⍳nodes]  ⍝ square matrix of nodes
    ∇

Now let’s wrap the SharpPlot initialisation into a single function that uses the .NET version if available and falls back to the cross-platform APL workspace if not:

    ∇ {dotnet}←Init
      :If dotnet←(,'W')≡3⊃'.'⎕WG'APLVersion'
          ⍝ .Net assembly (windows only)
          ⎕USING←',sharpplot.dll' ',system.drawing.dll'
      :Else
          ⍝ APL workspace (all platforms)
          :If 0=⎕NC'Causeway'   ⍝ copy workspace only once
              (System.Drawing←System←⍎'Causeway'⎕NS'').(⎕CY'sharpplot.dws')
          :EndIf 
      :EndIf
    ∇

Then we can write a platform-independent function that returns the SharpPlot instance with the chart drawn on it. We split nodes into two categories, functions and globals (ignoring locals and labels) and split links by destination node. By default, DrawNetworkMap lays out node categories on co-centric circles. Also, because the graph is uni-directional, we can afford to use straight links without damaging readability:

    ∇ sp←name Plot(mat nodes rows);catlabs;nodecat
      sp←⎕NEW Causeway.SharpPlot
     
      catlabs←'Function' 'Global'    ⍝ links are split by destination node
      nodecat←(≢nodes)⍴0             ⍝ ignored by default
      ((∨⌿mat∊'FR*')/nodecat)←1      ⍝ functions
      ((∨⌿mat∊'G')/nodecat)←2        ⍝ globals
      ((nodes∊rows)/nodecat)←1       ⍝ caller functions
     
      sp.Heading←name
      sp.HeadingStyle←Causeway.HeadingStyles.Left
      sp.SetMargins 40 30 30 30
      sp.KeyStyle←Causeway.KeyStyles.(BottomAlign+RightAlign)
      sp.SetILabels⊂nodes
      sp.SetILabelFont'Arial' 6 System.Drawing.FontStyle.Bold System.Drawing.Color.Black
      sp.SetColors⊂System.Drawing.Color.(SkyBlue LightCoral)
      sp.SetMarkers Causeway.Marker.Ball
      sp.SetMarkerScales 2
      sp.SetLineStyles Causeway.LineStyle.Solid
      sp.SetPenWidths 0.5
      sp.NetworkMapStyle←Causeway.NetworkMapStyles.(SplitByDestination+Dissected+ArrowLines+FixedArcs+NoAxes+NoLinkKey)
      sp.SetNetworkMapLinkArc 0         ⍝ straight lines
      sp.SetArrowStyle 5                ⍝ fixed-size arrows
     
      sp.SplitBy nodecat catlabs
      sp.DrawNetworkMap⊂↓(mat∊'GFR*')   ⍝ use a single width for all valid links
    ∇

We can then save the graph as SVG to a file (here we use the SvgMode that scales the SVG to fit its container):
sp.SaveSvg 'XRef.svg' Causeway.SvgMode.FixedAspect

Or, if feeding a web server or renderer, we can grab the SVG as a single string without writing to disk:
svg←sp.RenderSvg Causeway.SvgMode.FixedAspect

Windows users will be able to use the SharpPlot Viewer:
vw←⎕NEW Causeway.SharpPlotViewer sp
vw.Show ⍬

XRef

More examples of Network Maps can be found on the SharpPlot.com website in the Network Map Tutorials. To translate C# examples into APL, you can refer to the SharpPlot Rosetta Stone.

Over the Moon with Selenium

Over the years, I have become more and more reluctant to write code without good regression tests – 35 years of software development have taught me that it is just too dangerous! Regression testing of user interfaces has always been difficult, but for the last few weeks I have been working with technologies that put user interface testing within easy reach: HTML and the Selenium WebDriver, which star in the following video (if you can wait, read the full story below the video before watching):

This is exciting – I hope you can see why I am “over the moon” (apart from Selene being the Greek Moon Goddess).

Modern Markup Languages

One of the good things about modern User Interfaces based on markup languages like HTML is that logical information about the user interface is available at runtime. For example, a browser displaying HTML maintains a “Document Object Model” of the web page. The content and other attributes of each element within the DOM can be inspected and manipulated at runtime by a testing tool – this is what Selenium does.

Selenium

Selenium is a widely used open-source tool for automating browsers, with growing support from browser vendors. Selenium provides an API that allows you to perform just about any operation that a user could: click on buttons and select items from drop-downs, press keys to enter text, drag and drop items, or perform exact mouse movements and different types of clicks. Over the last few weeks, we have been working on a some thin covers for Selenium to make it easy to drive browsers from Dyalog APL. At the moment we require the Microsoft .NET bindings and can only do the testing from Microsoft Windows. The web server can be running anywhere, and we hope to add other client platforms in the future.

The results of this work are now available in the GitHub Repository Dyalog/Selenium. The following example, included in that repository, shows a simple test which verifies that the TryAPL website is able to execute a simple expression:

 ∇ r←Basic;S;result
 ⍝ Basic test that TryAPL is working - return error message if it isn't
 
 S←##.Selenium
 S.InitBrowser''
 S.GoTo'http://tryapl.org'
 
 'APLedit'S.SendKeys'1 2 3+4 5 6'
 S.('APLedit'SendKeys Keys.Return)
 result←'ClassName'S.Find'result'
 r←result S.WaitFor'5 7 9' '1 2 3+4 5 6 failed'
 ∇

MiServer Regression Testing

The primary motivation for this work has been to create regression test scripts for MiServer version 3.0. The MiServer regression testing mechanism is extremely simple (possibly a bit TOO simple, we’ll cross that bridge later): If a site contains a folder called “QA”, and the folders within mirror the structure of the web site, then a mechanism exists to load each page in turn and run the corresponding test function. You can read more about it on GitHub and see it working in the video at the top of this post.