Making the IDE My Own – Part 3 (Only Windows)

In Part 1, I built the machinery that runs a folder of small functions at startup, and in Part 2 I used that machinery to tune the Microsoft Windows IDE’s font, zoom, and keyboard shortcuts. Two extensions remain, and both are about information flowing in and out of the Session at a glance: one makes the title bar tell me exactly what I’m working with, and the other lets me drop things straight onto the Session.

What Am I Dealing With?

As mentioned in Part 1, the Session titlebar includes ⎕WSID by default, but setCaption solves another problem of situational awareness. I test various versions of many things, in particular, Link (I’m one of its main developers) and .NET (which Link uses to keep an eye on the file system). To distinguish my active interpreters, which often use disparate versions of Link and/or .NET, setCaption adds their version information to the Session titlebar while also abbreviating the interpreter characteristics:

∇ setCaption
 ⍝ Set Session caption to include Link and .NET versions
  ⎕SE.Caption←'{WSID} {NSID} - ',8↓¯1↓,(
      '(\d+\.\d+)[.\d]+'
      ' (\d*)(-bit)? (.)(nicode|lassic), BuildID \w+'
      '\.NET +\.NET( Framework)?'
      '  *'
  )⎕R(
      '\1'
      '\3\1'
      '.NET'
      ' '
  ),1 3 6⊂⍛⌷⎕SE.UCMD'TOOLS.Version'
∇

{WSID} and {NSID} are caption template fields that Dyalog keeps current in real-time, namely ⎕WSID (which autoLink sets to a fresh, timestamped working directory, as explained in Part 1) and ⎕THIS (the current namespace). After the dash comes the compact version string. I run ]TOOLS.Version, then 1 3 6⊂⍛⌷ plucks lines 1 (interpreter), 3 (Link), and 6 (.NET) out of the report.

An aside on ⊂⍛⌷: I usually want to select individual elements (major cells) rather than index along multiple axes. I’ve suggested adding this as a select primitive using the glyph ⊇; until that is implemented, to achieve this, the left argument of ⌷ needs to be enclosed. I have, therefore, learnt to read ⊂⍛⌷ as a single unit: select.

A small collection of regular expressions in one ⎕R then tidies the three lines – it strips the build ids, abbreviates “Unicode”/“Classic” to a single letter followed by the bits, squeezes runs of spaces, and collapses “.NET Framework” to “.NET” (“Framework” is implied if the version is 4.x):

Screenshot of Adám's Session showing lots of information in the titlebar.

My Session with lots of information in the title bar. Click to enlarge.

Drag, Drop, Done

I often want to import a function, some data, or even an entire source folder into the workspace. However, I was frustrated by the many ways in which this can be done:

  • ⎕FIX can read function/operator/object source files
  • ⎕CSV can read CSV (Comma-Separated Values) files, but needs variant options set for TSV (Tab-Separated Values), and so on
  • ⎕JSON can parse JSON (and with a variant option, JSON5), but cannot read files, so I have to first use ⎕NGET
  • ⎕XML can parse XML, but also needs ⎕NGET
  • ⎕SE.Dyalog.Array.Deserialise which handles array notation, as above
  • Link does source folders

In general, how a file should be imported is clear from its file extension (for example, .aplf, .csv, .tsv, .json, .json5, .xml, .apla), unless it is a folder. So, I wrote down all the file types I could think of with how I’d get their contents into Dyalog. That became the experimental ]Get user command, which I gave the slogan get pretty much anything from pretty much anywhere. Watch Rich Park’s webinar for an explanation of what it can do, or enter ]Get -?? in the Session for its documentation.

extStatusBar makes the status bar a drop target:

∇ extStatusBar
 ⍝ Extend Status bar to allow dropping files and folders
  ⎕SE.cbbot.bandsb1.sb.(AcceptFiles onDropFiles)←1 'Get'
∇

AcceptFiles←1 opts the status bar in, and onDropFiles←'Get' says which function to run when something lands on it:

∇ {msg}←Get(object event files item shift)
  :If 9≠40 ⎕ATX'get'
      {}⎕SE.UCMD'Get ]get'
  :EndIf
  ⎕←msg←# get.Get files,shift
∇

The function header decomposes the event message, although we only need two of the elements:

files
The list of dropped files and folders.
shift
The shifting state of the keyboard at the moment of the drop, encoded as a sum of 1 for Shift, 2 for Ctrl, and 4 for Alt (so 3 is Ctrl+Shift, and so on).

The first time the handler runs, the get namespace it relies on won’t be present – 40 ⎕ATX'get' won’t report 9 for “namespace” – so I fetch it by running the ]Get user command on itself! Thereafter, get.Get brings the dropped files into the workspace root (#). Fortunately, get.Get already used an encoded parameter, so I didn’t need to map the shifting state, just pass it on:

  • Holding Shift is equivalent to ]Get -sync: for example, a Link.Create rather than a plain Link.Import, which means that edits in the workspace are written back to disk (and vice versa where possible).
  • Holding Ctrl is ]Get -unpack: the contents are put directly into #, instead of into a namespace named after the dropped file or folder.
  • Alt does nothing yet, but I’m thinking of making it ask you how to import: ]Get allows a filename (or URL) to be followed by a colon and a file extension that overrides the existing extension. For example, ]Get data.csv:txt will import a CSV file as text rather than as a matrix. Let me know in a comment if you have a better idea!

With this, I can drop files, folders, and even zip files of data or code on the status bar, and everything simply appears in the workspace. Magical!

Screen recording showing drag-and-drop in action

Drag-and-drop of a data file, two code files, and a folder of code files.

Keep It Up!

That rounds out the series. In Part 1 I laid down the startup machinery, in Part 2 I tuned the font, zoom, and keys, and here I’ve made the title bar informative and the status bar a drop target.

In my next, and final, entry in this series, I’ll show you how to simplify some of this code using the new features of the recently released Dyalog v21.0. In the meantime, the easiest way to install everything is to paste all the function definitions from this series into a clear workspace and enter (with the directory that’s appropriate for you):

]LINK.Export # C:\Users\adam\Documents\Dyalog APL-64 20.0 Unicode Files\StartupSession\seext

Adding the next idea is a matter of writing a lowercase niladic function and dropping it in the folder (maybe together with some uppercase utilities for the functionality). And there is always a next idea; while writing these posts, I started thinking that I’d quite like to add a toolbar button that toggles ]Rows on and off…

Please comment if this was useful, and let me know if you have an idea for additional Session extensions!

Making the IDE My Own – Part 2 (Mostly Windows)

In my previous post, I laid the groundwork for extending the development environment by running a collection of small functions at startup time. If you’ve not read that post yet, I recommend doing so before you continue!

So far, my extensions have been generally applicable. I’ll move onto things that are specific to the Windows IDE, as that’s where I spend my days. That said, it might still be interesting to you even if you don’t use Dyalog on Microsoft Windows. For example, shortcut keys can be configured in Ride, and the font can be used in Ride and in your terminal emulator of choice.

A Font Face I Can Read

The first thing any APLer wants is a font that renders the glyphs properly and at a comfortable size. You might expect to set this in a configuration parameter – but you can’t; the session font and size are not configuration values. The traditional route is to choose them from the UI and then save the session file, but it is easy to automate making these choices every time we start. For the font, setFont drives the real controls, exactly as if I had used them with mouse and keyboard:

∇ setFont
 ⍝ Set font face and size
  :With ⎕SE.cbtop.bandtb5.tb
      font.Text←'APL387 Unicode'
      ⎕NQ font'Select'
      size.Value←17
      ⎕NQ size'Change'
  :EndWith
∇

⎕SE.cbtop.bandtb5.tb is the toolbar band that holds the font name combo (font) and the size spinner (size). For each control, you need to assign the new choice, then fire a Select event to make the change take effect.

Did you spot the 7 instead of 5? You might find it strange that I don’t use the traditional default APL385 Unicode font. I personally prefer the new APL387 Unicode, which was drafted by a professional type designer as a successor to the venerable APL385 Unicode. The new font is now being refined and the up-to-date revision is installed with every new build of Dyalog v20.0. By the way, Madeline Vergani was the very first applicant to receive a grant from The APL Trust, and it was for this project. It is, in my opinion, the cleanest our beloved glyphs have ever looked:

Screenshot comparing APL385 and APL387

Comparison of APL385 Unicode and the new APL387 Unicode.

In the above image, the first two lines below the titles illustrate APL387’s superior distinguishing of glyphs that could potentially be confused, the next line shows APL387’s more consistent design, and the final line shows examples of where APL387 just has nicer glyph shapes.

In Ride, you can set the font face permanently in the Preferences:

Screenshot of Ride's Preferences dialog showing how to change APL font

Ride: Preferences>Style>”APL font”

In and Out

setFont also settled on a size, but where the face is a set-and-forget choice, the ideal size depends on the situation. When I show my APL session during a video call or presentation, I usually need to increase my Session font size; conversely, I sometimes want a smaller font size to get an overview of a large piece of code, a large array, or a text-form diagram printed into the Session. These contradictory requirements mean that I like having font size controls at my fingertips:

∇ extViewMenu;ZC
 ⍝ Extend View menu with Zoom items
  ZC←{(⊃⍵)⎕WC'MenuItem'(2⊃⍵)('Event' ⋄ 'Select' 'Resize',3⊃⍵)('Hint' ⋄ 4⊃⍵)('Accelerator' ⋄ 5⊃⍵)}
  '⎕SE.mb.view.sep'⎕WC'Separator'
  ZC¨(
      '⎕SE.mb.view.larger'  'Zoom &In'   1 'Make text larger (Ctrl+=)'  (187 2)
      '⎕SE.mb.view.smaller' 'Zoom &Out' ¯1 'Make text smaller (Ctrl+-)' (189 2)
  )
∇

ZC (for Zoom Create) is a small helper function that removes repetitive parts of the code. When the event specification has a third element after the event name and callback function, like 'Select' 'Resize' 1, the third element is used as the left argument to the callback function. That function is Resize in our case, which, like setFont, works on the native control. It uses the size spinner’s own list of sizes rather than guessing at point values:

∇ a Resize e
 ⍝ Resize Session font
  ;font;idx;new;sizes
  font←⎕SE ⎕WG'Font'
  sizes←⎕SE.cbtop.bandtb5.tb.size.Items
 
  idx←a+sizes⍸⍥{⌈⍵÷2}2⊃font
  new←sizes⊃⍨1⌈idx⌊≢sizes
 
  ⎕SE.cbtop.bandtb5.tb.size.(Value Thumb)←new idx
  ⎕NQ ⎕SE.cbtop.bandtb5.tb.size'Spin'idx
∇

This locates the current size’s location (with fuzzing) within the spinner’s Items, takes one step in the requested direction (while protecting against leaving the range), and writes it back into the control, then fires a Spin event so the session re-renders. Because the control owns the list of legal sizes, I can never land on one it wouldn’t have offered. Anyway, this is what my View menu looks like now:

Screenshot of Adám's View menu after running extViewMenu

What my View menu looks like after running extViewMenu

⎕PFKEY doesn’t reach Ride, but Ride’s own Preferences>Shortcuts accepts many of the same command codes if you’d like to reproduce the effect there; it allows you to assign any keystroke to an “F-key” too. (However, you don’t need to go through F-keys to have two keystrokes for a single action; Ride allows assigning multiple keystrokes to each action.) Ride also already has keyboard shortcuts for zooming in and out, and resetting the zoom:

Screenshot of Ride's Preferences dialog showing how to change font size

Ride: Preferences>Shortcuts>”font”

Function Keys Revisited

In 2018 I employed ⎕PFKEY to let the Tracer run up to a chosen line: I click on the line I’m interested in, which moves the text caret there, and the Tracer then executes up to that line and stops — a quick way to skip ahead to the part I actually want to inspect. In the 2020 follow-up I added the ability to select (part of) an expression in the Editor and have it executed in the session without (apparently) leaving the Editor. Both of those keys live on in setFKeys, alongside a good many friends (ignore the keystrokes mentioned in the comments for now – I’ll explain those in the next section):

∇ setFKeys
 ⍝ Set various F-keys
  ;PF
  PF←{}⊆⍛⎕PFKEY⍨
 
  5 PF' ' 'DB'                  ⍝ Mark for execution   F5
  6 PF'CP' 'JP' 'PT' 'ER' 'JP'  ⍝ Run selection in session  F6
 
  13 PF')' 'LL' '(' 'RL'  ⍝ Parenthesise left  Ctrl+0
  14 PF']' 'LL' '[' 'RL'  ⍝ Bracket left       Ctrl+]
  15 PF'}' 'LL' '{' 'RL'  ⍝ Brace left         Ctrl+}
 
  10 PF'BP' 'RM' 'TC' 'BP'       ⍝ Trace to here    F10
  34 PF'BP' 'RM' 'TC' 'BP' 'TC'  ⍝ Trace into here  Ctrl+F10
 
  43 PF'OP' 'IL'       ⍝ Insert open line below  Ctrl+I
  44 PF'ER' 'BK' 'RL'  ⍝ Execute and recall      Alt+Enter
 
  45 PF'BK'  ⍝ Ctrl+Z
  46 PF'FD'  ⍝ Ctrl+Y
 
  47 PF'Lw' 'DI'  ⍝ Delete word left   Ctrl+BkSp
  48 PF'Rw' 'DI'  ⍝ Delete word right  Ctrl+Del
∇

A word on the numbers: ⎕PFKEY programs numbered Programmable Function keys, but only some of them are literally function keys. So, while, for example, 5 and 6 are F5 and F6, numbers above 12 refer to shifted keys:

  • 13 through 24 are Shift+F1 through F12
  • 25 through 36 are Ctrl+F1 through F12
  • 37 through 48 are Ctrl+Shift+F1 through F12
  • 48 through 96 are as above, but while holding Alt

The whole function is held together by the first line: ⎕PFKEY takes the shortcut keys and symbols on the left and the F-key number on the right. However, I prefer to write the number on the left, so I assemble a tiny tacit function:

  • Commute (⍨) flips the arguments so I can give the number on the left.
  • Behind (⍛) allows nest (⊆) to ensure that the (new) left argument (the shortcut keys) is nested. This is so that a single code like 'BK' is treated as if I’d written ⊂'BK' – a small notational nicety that lets the single-code keys read as cleanly as the rest.
  • The leading {} swallows the returned result (the previous value for the F-key) so that the function is silent.

In Ride, some of these come pre-assigned. Many others can be constructed by typing key sequences into the input fields for PFn entries under Preferences>Shortcuts, where you can also assign any keyboard combination:

Screenshot of Ride's Preferences dialog showing how to change F-keys

Ride: Preferences>Shortcuts: “PF”

Keyboard Shortcuts

Options>Configure…>Keyboard Shortcuts allows us to assign custom shortcuts for the “keys”, up to number 48, to something more comfortable. That’s what I do with 13, 14, and 15; reassigning them to Ctrl+0, Ctrl+], and Ctrl+} for the three “wrap” keys (parenthesise/bracket/brace from here to the left), Alt+Enter to execute the current line and recall it for further tweaks (good for teaching), and Ctrl+Backspace and Ctrl+Delete to delete a word left or right (these are virtually a standard, but Dyalog’s IDE doesn’t have them as available commands).

Why am I assigning F-keys 45 and 46 to <BK> and <FD> when these actions can already be given keyboard shortcuts? Because they serve double (actually, triple!) duty: They both navigate the Session input history (and skip backwards/forwards in the Tracer) and Undo/Redo edits. In my head, those are completely different concepts, and everywhere else I use Ctrl+Z and Ctrl+Y for Undo and Redo, so I’d like to use them as such in Dyalog too — while simultaneously keeping the default keystrokes Ctrl+Shift+Backspace and Ctrl+Shift+Enter for navigating Session history and lines in the Tracer. By way of an unused F-key, I can have two keystrokes for the same shortcut!

Screenshot of Adám's Preferences window, showing many of the shortcuts involving the Ctrl key

My Preferences window, showing many of my shortcuts that use the Ctrl key.

To Be Continued…

So far in this post, I have tuned the font, put zoom controls within reach, and reprogrammed a batch of keys. In the next post of this series, I’ll enhance the Windows IDE title bar and status bar. In the meantime, the easiest way to install everything is to paste all the function definitions from this series into a clear workspace and enter (with the directory that’s appropriate for you):

]LINK.Export # C:\Users\adam\Documents\Dyalog APL-64 20.0 Unicode Files\StartupSession\seext

If any of this is useful to you, take it. If it doesn’t quite do what you want, adapt it. If you have an idea for additional Session extensions, let me know!

Hashing It Out: Lookup Performance in Dyalog APL

Sometimes it seems that I have the attention span of a cocker spaniel puppy. I start to look at one thing, and then something pops up to distract me and I follow that until another distraction or tangent comes along and then…. well, you get the point. What started out as one blog post has morphed a few times into what you’re about to read!

Background and Original Goal

In a former life, I wrote an APL-based full-text search engine for searching legislative and regulatory texts. At the time, its speed rivalled any of the online search engines. In that governmental context, words like “the” and “of” have significance, for example, the “the” in “the white house” is significant, as are the “of” and “the” in “speaker of the house”. Most search engines would ignore those high-frequency “noise” words – mine couldn’t. My goal was to be able to search for such phrases without having to examine the data for the high-frequency words. I found this to be an interesting problem at the time (in the 1990s) and subsequently included it 25 years later as the 2015 APL Problem Solving Competition Phase 2 Applications Problem 2. My original idea for this blog post was to examine approaches to that problem.

The core of the problem was straightforward – you have a list of words (vector of character vectors) and a corresponding vector of frequencies for each word. We’ll call these two vectors Words and Freqs. For those interested, the source data I used for my recent effort came from a repository that tracks Wikipedia word frequencies. The task is to, for a given phrase, break the phrase up into individual words (we’ll call this words), look up words in Words, and use the indices to extract their frequencies from Freqs. This is basic APL, the sort of thing we do all the time: Freqs[Words ⍳ words]. If we want to consider the case where a word in words isn’t found in Words, we can append 0 to Freqs.

The source for Words and Freqs has 2,765,377 words. To emulate my original work done in the 1990s when we were using only ASCII characters, I decided to remove words containing non-ASCII characters. As a result, we’re left with 1,905,526 words.

Tangent #1 – ⎕CSV

If you’re interested in obtaining Words and Freqs yourself, you can download the raw data file and use ⎕CSV and a bit of code:

      (Words Freqs)←⎕CSV⍠('Separator' ' ')('Invert' 2)⊢'{your-path-here}/enwiki-2023-04-13.txt' ''(1 2)
      (Words Freqs)/⍨←⊂Words(∧/∊)¨⊂⎕C⎕A ⍝ limit words to ASCII alphabetics

⎕CSV can do some amazing things. Adám Brudzewsky made a very interesting video on Parsing content from text files using ⎕CSV.

From that list, the five most and least frequent words are:

      ⍉↑5(↑,-⍛↑)¨Words Freqs ⍝ look mom! I used ⍛ (behind)!
 the          186631452
 of            88349543
 in            76718795
 and           76039670
 a             54631147
 byodoinji            3
 houryuuji            3
 groendorpen          3
 witotoanas           3
 kimely               3

Interestingly, it seems a word must occur at least 3 times in Wikipedia to make it into this list. Okay, enough background information…

Distraction #1

In my testing, I found that, when words exceeded 3 elements, things slowed down a lot. For example:

      ]RunTime -c "Words⍳'the' 'big' 'dog'" "Words⍳'the' 'big' 'fuzzy' 'dog'"
  Words⍳'the' 'big' 'dog'         → 1.3E¯4 |       0%                                          
* Words⍳'the' 'big' 'fuzzy' 'dog' → 2.5E¯1 | +202400% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 

One would expect the performance to degrade linearly with the number of words. This appears to be true until you reach 4 elements in the right argument. Let’s create some test cases using lists of 1 to 6 words; since ‘the’ is the first word in the list, these searches should be as fast as possible.

      (p1 p2 p3 p4 p5 p6)←1 2 3 4 5 6⍴¨⊂'the'

Now let’s see what happens when we search for lists of 1 to 4 words:

      ]RunTime -c Words⍳p1 Words⍳p2 Words⍳p3 Words⍳p4
  Words⍳p1 → 2.1E¯2 |     0% ⎕⎕⎕                                      
* Words⍳p2 → 4.2E¯2 |  +102% ⎕⎕⎕⎕⎕⎕                                   
* Words⍳p3 → 6.2E¯2 |  +204% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕                               
* Words⍳p4 → 2.6E¯1 | +1147% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 

It looks linear for lengths 1 to 3, but jumps at length 4. If we look at lengths 4 to 6, we see a linear pattern, albeit considerably slower.

      ]RunTime -c Words⍳p4 Words⍳p5 Words⍳p6
  Words⍳p4 → 2.5E¯1 |  0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕   
* Words⍳p5 → 2.5E¯1 | +1% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕  
* Words⍳p6 → 2.6E¯1 | +4% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 

What’s going on?! After a bit of research (I asked Morten!), I found that Dyalog is hashing Words extemporaneously when words exceeds 3 elements. Because the hash table is not retained, Words is hashed for each iteration in ]RunTime, causing the slowdown.

Since creating the hash table is a relatively expensive operation, it only makes sense to do if there is a mechanism to retain the table from which subsequent searches can benefit or the right argument is long enough to justify the upfront cost.

Tangent #2 – A Brief History of Hashing in Dyalog APL

First, a bit of terminology – search functions have a “principal” argument (the array being searched) and a “subject” argument (the items being searched for). In the example above, Words is the principal argument and words is the subject argument.

Hashing, in one form or another, has existed in Dyalog APL from its earliest days; it’s an industry standard technique. The interpreter decides, based on a number of factors, whether to hash an array extemporaneously or not. This hashing is an internal process and beyond the user’s control.

Dyalog v10.0, released in 2003, gave users the ability to bind a search function like ⍳ (index of), ∊ (membership), ∩ (intersection), ∪ (union), or ~ (without) to its principal argument, resulting in a derived function that will, on its first invocation, create and retain a hash table. The disadvantage of this is that the hash table is used exclusively for the bound search function. If you needed to use more than one search function you would bind them individually and each would have its own hash table.

Dyalog v15.0, released in 2016, introduced 1500⌶ (hash array). This allows the user to mark an array for hashing and, after the first search operation creates the hash table, the hash table is available for all applicable search functions. array←1500⌶array returns a copy of array that is marked for hashing. The hash is created on the first invocation of a search function on array. 1(1500⌶)array returns 0 if array has not been marked for hashing, 1 if array has been marked for hashing but the hash has not been created yet, and 2 if array has a hash table. For example:

      1 (1500⌶) Words ⍝ not marked yet
0
      Words←1500⌶ Words ⍝ mark for hashing
      1 (1500⌶) Words
1
      Words⍳⊂'the' ⍝ perform a search (this is a bit slow)
      1 (1500⌶) Words ⍝ Words now has a hash table
2
      Words,←⊂'thermoflocker' ⍝ add a new word to the tail end
      1 (1500⌶) Words ⍝ still hashed
2
      Words← 1 ↓ Words ⍝ drop something from the front end
      1 (1500⌶) Words ⍝ no longer hashed nor marked for hashing
0

The size of the hash table can impact performance. In general, larger hash tables result in faster lookups. Dyalog v19.0, released in 2024, introduced 8468⌶ (hash table size), which allowed the user to adjust the size factor of created hash tables. Its purpose was to allow users to evaluate the potential side effects of a proposed larger hash table size.

Hash tables in Dyalog APL work best with (mostly) static principal arrays. In general, modifying the array invalidates the hash and the array will need to be rehashed. However, there are three forms of modified assignment that will preserve and efficiently update the hash – notice that they all involve the tail end of the principal array.

      R,←Y    ⍝ only for scalar or vector R
      R⍪←Y
      R↓⍨←Y   ⍝ only for negative singleton Y

Dyalog v20.0, released in 2025, implemented hash tables that are 8 times (2*3) larger than previously and also removed 8468⌶. More on this later…

Why Hash?

The simplistic approach to search functions like ⍳ and ∊ is to perform a linear search, starting at the first element of the principal argument and iterating through its elements until you find a match. Obviously, elements that occur near the front will be found faster. In a worst case, searching for an element that’s not in the principal argument will have to search its entirety. Doing a linear search on the first word in Words and for a word not found in Words demonstrates this:

      ]RunTime -c "Words⍳⊂'the'" "Words⍳⊂'thermoflocker'"
  Words⍳⊂'the'           → 0.0E0  |    -100%                                          
* Words⍳⊂'thermoflocker' → 2.0E¯2 | +250200% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 

Hashing smooths out this variability and makes all lookups relatively similar in performance, not to mention considerably faster. Doing the same search on aWords, a hashed version of Words, shows this.

      ]RunTime -c "aWords⍳⊂'the'" "aWords⍳⊂'thermoflocker'"
  aWords⍳⊂'the'           → 2.1E¯7 |   0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕     
* aWords⍳⊂'thermoflocker' → 2.4E¯7 | +11% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 

“I Feel the Need for Speed” (Top Gun, 1986)

Let’s revisit the issue that led us down the hashing path. We’ll create a hashed copy of Words, but leave Words alone so that we can compare performance. We’ll also create a function bound with ⍳.

      hWords←1500⌶Words ⍝ hWords is marked for hashing
      hWords⍳⊂'the'      ⍝ first search creates the hash table
1
      fWords←Words∘⍳      ⍝ also create the derived function  
      fWords ⊂'the'      ⍝ first search creates the hash table
1
      p←'the' 'big' 'fuzzy' 'dog'
      ]RunTime -c "hWords ⍳ p" "fWords p" "Words ⍳ p"
                                                                        
  hWords ⍳ p → 0.0E0  |    -100%                                          
  fWords p → 0.0E0    |    -100%                                          
  Words ⍳ p  → 3.5E¯1 | +556800% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 

Once we have a retained hash table, all lookups, no matter the length of words, will be faster.

Tangent #3 – When Does a Principal Argument Get Hashed?

The short answer is: it depends on a number of factors, such as:

  • whether a search function like ⍳ or ∊ is used
  • the sizes of the principal and subject arrays
  • the datatypes of the arrays – character, integer, floating point, complex
  • are the arrays simple (depth 0 or 1) or not?
  • the amount of workspace available – can it fit the hash table?

In short, Dyalog APL hashes an array when it “thinks” doing so will help. However, you know your application and its data better than the interpreter does. In my example above, the interpreter can’t know if my search is a one-time thing or something that will be executed many times.

How can you determine where hashing kicks in? What I did was to start with a single element subject argument and increase its length until I saw a “bump” in runtime:

      iv←1e7?2*24  ⍝ build 10-million element integer vector with no duplicates

      ⍝ now time lookups for subject arrays of length 1 to 15
      ⎕SE.UCMD 'runtime -c',∊' iv⍳iv['∘,¨(⍕¨⍳15),¨⊂'⍴5000000]'
  iv⍳iv[1⍴5000000]  → 1.0E¯3 |     0%                                          
* iv⍳iv[2⍴5000000]  → 2.1E¯3 |  +100% ⎕                                        
* iv⍳iv[3⍴5000000]  → 3.2E¯3 |  +206% ⎕                                        
* iv⍳iv[4⍴5000000]  → 4.3E¯3 |  +315% ⎕⎕                                       
* iv⍳iv[5⍴5000000]  → 5.3E¯3 |  +412% ⎕⎕                                       
* iv⍳iv[6⍴5000000]  → 6.2E¯3 |  +503% ⎕⎕⎕                                      
* iv⍳iv[7⍴5000000]  → 7.3E¯3 |  +609% ⎕⎕⎕                                      
* iv⍳iv[8⍴5000000]  → 8.4E¯3 |  +718% ⎕⎕⎕⎕                                     
* iv⍳iv[9⍴5000000]  → 9.3E¯3 |  +800% ⎕⎕⎕⎕                                     
* iv⍳iv[10⍴5000000] → 1.0E¯2 |  +909% ⎕⎕⎕⎕⎕                                    
* iv⍳iv[11⍴5000000] → 9.2E¯2 | +8784% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 
* iv⍳iv[12⍴5000000] → 9.1E¯2 | +8757% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 
* iv⍳iv[13⍴5000000] → 9.2E¯2 | +8818% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 
* iv⍳iv[14⍴5000000] → 9.2E¯2 | +8821% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 
* iv⍳iv[15⍴5000000] → 9.2E¯2 | +8806% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕ 

Can you spot where hashing kicks in? That’s right – when the subject array exceeds 10 elements. But what if my subject array is only ever a single element? Then you need to determine whether the one-time cost of building the hash table is less than the accumulated time of subsequent lookups. Once hashed, all lookups are faster.

One caveat about using the ]RunTime user command – the cost of creating the hash table on the first lookup of a hashed array is lost because ]RunTime executes each expression once before collecting timing information. ⎕PROFILE provides a more accurate picture:

     ∇ r←test_iv;iv;hiv;_;d;l;t;i
[1]    iv←10000000?2*24 ⍝ 10,000,000 integers in the range 1-2*24
[2]    hiv←1500⌶iv      ⍝ hiv is marked for hashing
[3]    i←5000000⊃iv     ⍝ search for the middle element
[4]   l1:⎕PROFILE¨'stop' 'clear' 'start'
[5]    _←iv⍳i  ⍝ unhashed lookup
[6]    _←hiv⍳i ⍝ first hashed lookup (creates hash table)
[7]    _←hiv⍳i ⍝ subsequent hashed lookup     
[8]   l2:⎕PROFILE'stop'
[9]    d←⎕PROFILE'data'
[10]   l←l1+⍳l2-l1+1 
[11]   r←(↓'%',⍨⍕⍪⌊0.5+100ׯ1+t÷⌊/t),(⎕NR⊃⎕SI)[1+l],⍨3⍕⍪t←d[;4]⌿⍨d[;2]∊l
[12]   ⎕PROFILE'clear'
     ∇

      test_iv
   1044%   1.047  _←iv⍳i  ⍝ unhashed lookup
 236125% 216.147  _←hiv⍳i ⍝ first hashed lookup (creates hash table) 
      0%   0.092  _←hiv⍳i ⍝ subsequent hashed lookup

In this case, the cost of creating the hash table is about 200× the cost of doing a single lookup. Subsequent hashed lookups are about 10× faster. It’s up to you, and your understanding of your application, to determine whether retaining the hash table is worth it.

TANSTAAFL (There Ain’t No Such Thing As A Free Lunch)

The other cost of using hash tables for faster searching is that hash tables take up space. Remember when I mentioned that the default hash table size in Dyalog v20.0 is 8x larger than previously? The working theory is that larger hash tables ought to result in faster searches, but how much faster and at what cost in space? Unfortunately there isn’t a direct way to measure the hash table size. You have to look at the differential in ⎕WA before and after creating the hash table:

      ]Config MAXWS
 MAXWS  1GB
       
      cf ⎕SIZE'Words' ⍝ cf is a utility to comma-format numbers
97,547,968

      cf ⎕WA-⍨(hWords⍳⊂'the')⊢(hWords←1500⌶Words)⊢⎕WA
268,435,664

In my 1GB workspace, the hash table for Words is 268MB and 2.75× the size of the Words itself. In Dyalog v20.0, we removed 8468⌶ which would allow you to set a scaling factor for the hash table size; there is an experimental I-Beam currently being developed, 8470⌶1 which does much the same thing. 8470⌶1 factor, where factor is an integer in the range ¯3 to 3, will increase or decrease the hash table size by 2*factor times. When Dyalog v20.0 was initially released, the default value for factor was 0 but based on feedback it’s now ¯3 Let’s use this to examine its effect on hash table size. I wrote an ugly little operator, set, to build hash tables with specified scaling factors and look at the ⎕WA differential after each:

      ]Config MAXWS
 MAXWS  10GB
      'abcdefg' ('Words'set) ¯4+⍳7
 aWords  ¯3       33,555,104 
 bWords  ¯2       67,109,536 
 cWords  ¯1      134,218,400 
 dWords   0      268,436,128 
 eWords   1      536,871,584 
 fWords   2    1,073,742,496 
 gWords   3    2,147,484,320 

As expected, each hash table grows by a factor of 2.

Tangent #4 – Mapped Files

The ⎕MAP system function associates a mapped file with an array in the workspace. One of the client projects I’ve worked on involves loading CSV data with 47 million records of 85 fields, totalling about 7.5GB. Each field is stored in a separate ⎕MAP-compatible file. In a 4GB workspace, I’m able to map all 85 files and have it seemingly consume only a bit over 2MB of workspace:

      ]Config MAXWS
 MAXWS  4GB 

      ⍝ MapFiles maps 85 files into variables in a namespace named Data
      cf wa←⎕WA ⋄ MapFiles ⋄ cf wa-⎕WA 
4,284,076,808
2,009,600

      cf +/Data.(⎕SIZE ⎕NL ¯2)
7,577,998,192

Pretty neat huh? This makes 7.5GB of data available in a 4GB workspace!

One of the fields, Data.Record_ID, is a 16-column character matrix and has a size of about 760MB:

      ⍴Data.Record_ID
47647772 16
      cf ⎕SIZE 'Data.Record_ID'
762,364,392

Let’s bind ⍳ with Data.Record_ID:

      lFind←Data.Record_ID∘⍳     ⍝ create the bound function
      lFind 1↑Data.Record_ID     ⍝ create the hash table
WS FULL
      fRec 1↑Data.Record_ID
      ∧

Uh oh… It seems we don’t have enough room in a 4GB workspace for the hash table. Let’s start a new session with 10GB:

      ]Config MAXWS
 MAXWS  10GB 
      lFind←Data.Record_ID∘⍳
      cf ⎕WA-⍨(lFind 1↑Data.Record_ID)⊢⎕WA
8,589,934,752

With the Dyalog v20.0 default scaling factor, the hash table takes 8.5GB of my 10GB workspace. That’s a hefty chunk. What if we try the smallest scaling factor, ¯3?

      8470⌶1 ¯3
0
      sFind←Data.Record_ID∘⍳
      cf ⎕WA-⍨(sFind 1↑Data.Record_ID)⊢⎕WA
 1,073,741,984

That’s more manageable and would fit in my 4GB workspace. Why did I use a bound function rather than 1500⌶ on Data.Record_ID? Currently, 1500⌶ doesn’t work with mapped files, but this may be taken under consideration in the future.

      z←1500⌶Data.Record_ID
DOMAIN ERROR
      z←1500⌶Data.Record_ID
            ∧

Tangent #5 – What Timer Is It?

The ]runTime user command is a wonderfully easy tool to use to get a rough idea of performance and it remains a staple in my toolbox. runtime is based on cmpx which is found in the dfns workspace. I’ve found that runtime‘s results can sometimes vary significantly from one invocation to the next. Dyalog APL provides several system functions to help collect performance data:

  • ⎕AI, one of whose elements is cumulative CPU time of your session. You can use the before and after differential of ⎕AI to get an approximate idea of the speed of an expression. This is what runtime and cmpx use.
  • ⎕MONITOR, which can be used to turn monitoring on for specific lines of a function.
  • ⎕PROFILE, the “heavy hitter” of performance measurement. Even though its primary use case is profiling the performance of entire applications, I still find it useful to get impressive-looking, high-precision, performance data.

I have a template that I use when I want to use ⎕PROFILE to compare performance of expressions:

     ∇ r←timer n;i;d;l;t
[1]   ⍝ n is the number of iterations to run
[2]   ⍝ r is [;1] the relative percents based on the fastest expression
[3]   ⍝      [;2] the accumulated time in ms for the expression
[4]   ⍝      [;3] the expression
[5]    ⎕PROFILE¨'stop' 'clear' 'start'
[6]   l1: :For i :In ⍳n
[7]   ⍝ insert expressions to time between l1 and l2
[8]   l2: :EndFor
[9]    ⎕PROFILE'stop'
[10]   d←⎕PROFILE'data'
[11]   l←l1+⍳l2-l1+1
[12]   r←(↓'%',⍨⍕⍪⌊0.5+100ׯ1+t÷⌊/t),({⍵↓⍨+/∧\' '=⍵}¨(⎕NR⊃⎕SI)[1+l]),[1.1]⍨↓3⍕⍪t←d[;4]⌿⍨d[;2]∊l
[13]   ⎕PROFILE'clear'
     ∇

The Speed/Space Tradeoff

We saw earlier that different scaling factors result in different size hash tables, but what’s the impact on the speed of searching? Using our 7 hashed versions of Words and running 1,000,000 iterations:

 9%   547.303  _←aWords⍳phrase 
 6%   534.984  _←bWords⍳phrase 
 0%   504.180  _←cWords⍳phrase 
 0%   504.822  _←dWords⍳phrase 
 0%   505.082  _←eWords⍳phrase 
 0%   505.404  _←fWords⍳phrase 
 0%   504.561  _←gWords⍳phrase 

In this example, the smallest hash table, aWords runs 9% slower than the 8-times-larger default, dWords. There’s no significant speedup using a hash table larger than cWords, at half the default size.

Let’s take a look at my 47-million record table. Since 1500⌶ doesn’t work (yet) with mapped files, we’ll have to use function binding. First, we’ll use the default setting of 8470⌶1 0 to create an 8.5GB hash table and then use 8470⌶1 ¯3 to create a hash table 1/8 (2*¯3) the default size or about 1GB.

      {}8470⌶1 0 ⋄ b←⎕WA ⋄ lfind←Data.Record_ID∘⍳ ⋄ {}lfind 2↑Data.Record_ID ⋄ cf b-⎕WA
  8,589,934,808
      {}8470⌶1 ¯3 ⋄ b←⎕WA ⋄ sfind←Data.Record_ID∘⍳ ⋄ {}sfind 2↑Data.Record_ID ⋄ cf b-⎕WA
  1,073,742,040
      recs←Data.Record_ID[10?≢Data.Record_ID;]
      timer 1e6
  0%    920.916  _←lfind recs 
 11%   1025.790  _←sfind recs 

Is an 11% speedup worth an extra 7.5GB of workspace? That’s really up to you and your priorities for your application.

The Long and Winding Blog (with apologies to The Beatles)

Let’s try to summarize…

If your application needs to search data that’s “mostly” static, you can benefit from using retained hash tables. By “mostly”, I mean that the only changes to the array are adding or deleting elements from the end. A typical scenario might be, during application initialization, load the data into the workspace, or map a file, and do a search, thereby creating the hash table. Then all subsequent uses of search functions on that array will be considerably faster.

I’ve got some ideas on things that we at Dyalog Ltd might want to consider doing. Most of these involve empowering the user to have more control over how and when hashing is done. For example, we could:

  • make 8470⌶1 ready for release and publish it so that users can understand how to tune hash table size for their best space/speed tradeoff.
  • make 1500⌶ work with mapped files.
  • make it easier to query the size of a hash table rather than relying on ⎕WA differentials.
  • make it possible to mark an array as “unhashable”. Consider the scenario where hash-table-triggering searches are done on an array that’s not static. Whenever the array changes, the hash table is discarded, only for a new hash table to be generated on a subsequent search. In this case, it’s probably better that the interpreter fall back to a non-hashing approach.
  • Make it possible to monitor where, when, and how often extemporaneous hashing is done.

Disclaimers – YMMV (Your Mileage May Vary)

The examples shown in this post are a very small sampling of possible hashing scenarios. They were run on my 32GB Lenovo laptop using Windows 11 and Dyalog v20.0. Your environment and hashing opportunities are likely to be different. In writing this post, I hope to have accomplished spurring you on to do your own investigations as there are potentially significant benefits to be explored.

I was caught off guard by the apparent slowdown due to the interpreter repeatedly hashing Words extemporaneously. This led me to try to understand how and when hashing is done and what tools are available to the user. I feel I’ve been marginally successful at this, but it feels like there is still much to learn.

Addendum

I wrote the above in January 2026. Sharing my findings with the Dyalog development team set in motion an initiative to review some aspects of doing lookups. One result of this initiative was to change the default scaling factor for hash table size (8470⌶1 x) from 0 to ¯3. This means that, by default, hash tables in the latest releases of Dyalog v20.0 are 1/8 the size that they were before. Changing to a smaller hash table scaling factor results in slightly slower lookups, but at a significant savings in workspace consumed.

Other work looking at the performance of lookup functions is being undertaken.

Lookups are Complicated

When you use a lookup function like ⍳ or ∊, the interpreter attempts to pick the best technique, in order from simplest to most complex, between doing a linear search, using a lookup table, or using a hash table. The decision is based on datatype, size, and depth of the arguments, workspace available, and whether the cost of doing the setup of a more complex technique is likely to result in faster overall execution. In general, the interpreter does a good job of deciding. That doesn’t mean there aren’t exceptions. Again, this is where you come in – you know your data and how it’s used.

Employee Spotlight: Martin

When Martin joined Dyalog last year, he confessed to a secret identity… beneath the formal attire hides a medieval merchant! Twelve months later, we’ve noticed the resemblance runs deeper than the tunic. Martin is, in fact, a bit like Batman. He comes and goes at odd hours, responds best when there’s a symbol in the sky (or a ticket in the Technical Services queue!), and quietly solves problems wherever he lands, from cable management in the Copenhagen office to overhauling Dyalog Ltd’s internal IT infrastructure. His one true nemesis remains the office printer, which continues to resist all attempts at diplomacy. Replacing it may be next year’s headline project.

Since arriving, Martin has taken on the behind-the-scenes work that makes everyone else’s day run smoother, from setting up home offices to gathering the Technical Services team for their first conclave on collaboration. That variety is exactly what he enjoys most. “I do everything IT-related. I hadn’t changed printer toner or installed a router since I was a trainee, so getting to mix that with writing communication policies and migrating to Microsoft Exchange has been a lot of fun.”

Coming from more corporate environments, Martin found Dyalog Ltd’s culture to be a refreshing change of thinking. “Everyone here is extremely passionate about what they do, whether that’s developing new features, keeping the IT environment healthy, or making the business run smoothly. That energy is contagious, and it’s something I didn’t know I was missing.” What struck him most was the absence of a career ladder to climb or internal competition to navigate. Instead, there’s a flat hierarchy and an open culture where everyone’s opinion matters.

“I feel at home whether I’m in the office or working remotely. It’s very much the company culture, and one of the things I enjoy most about working here. Being based in Denmark, most of Martin’s collaboration happens with the team in Bramley, an arrangement that suits him nicely. “Nine-to-five has never really been for me. I’m a bit of a night owl, so being able to come in late and stay late has genuinely improved my wellbeing.”

Life outside the office has scaled up too. Over the past year, Martin has expanded his Merchant’s Guild. He now works more than one market at a time, and traded his central Copenhagen apartment for a farmhourse, a plot of land, and several old stables. So, when Martin is “on leave,” he is definitely not relaxing with a drink in hand but rather is out fixing the quirks of an old farmhouse, entertaining market guests, and adding to the general, glorious, chaos of it all.

After his first twelve months, his guiding philosophy hasn’t changed. “Technology should make life easier, not more complicated. I can’t develop features for the interpreter or decide the direction for APL, but if I can make the developers’ lives a little easier so that they can focus on what they’re brilliant at, then that’s success for me!”

The first year has gone by in a whirl, and Martin is looking forward to many more, plenty of projects to dive into, lots of debates to be had, and a Copenhagen office fridge full of Coke that is unlikely to empty itself.

Working with LLMs and Dyalog

Notes about this blog post:

  • In this blog post, I’m using macOS and Docker. If you’re a Microsoft Windows user and you want to follow along, Docker works well on WSL2.
  • AI is a fast-moving field, and I’ll assume some familiarity with LLMs in terms of terminology.

Working with an AI coding “agent” can make us more productive by automating boiler plating, helping with testing, and so on. LLMs are very good at languages like Python and C#, but have struggled with APL. We can speculate as to why this is, but the answer is most likely as mundane as the lack of APL “out there”. The training sets are sparse when it comes to APL. The other aspect is one of incentives: no frontier AI laboratory has any real incentive to make their models better at APL; they’re usually Python shops.

However, the pace of improvement in LLMs generally, and the drive towards “smarter, not bigger” models, now make LLMs viable as a productivity tool when working with Dyalog APL. The latest models from Anthropic, OpenAI, and Google today write passable APL – still a bit naive and Python-like, but capable of writing functional, non-trivial APL. In this blog post, I will outline my working practices and set-up, and a few practical tips on how to make an LLM more fluent in APL.

Warning: using an AI coding agent requires care. Although running an AI agent in a container narrows the blast radius, many risks remain unavoidable, especially when running in more autonomous modes.

“The November 2025 Inflection Point”

Two model releases happened in November, 2025 – Claude Opus 4.5 and GPT-5.2 – within days of each other. Simon Willison coined the expression The November 2025 Inflection Point for this quantum leap. Up until then, LLM performance on APL was abysmal: they were unable to understand the right-to-left execution order and really struggled with the array model in general, let alone syntax. After the November release, all of that changed. Although the models were still not exactly fluent in APL, it was a real step change, especially in their ability to explain APL code. APL performance is lifted by the general improvement in models over time – it remains far behind the performance in more mainstream languages, but is now something that is approaching useful to APL programmers.

Tooling Improvements

In conjunction with the model improvements, suddenly “Agents” took off. How we interact with LLMs has also evolved, from the original CoPilot smart auto-complete, to copy-pasting code into the ChatGPT web interface, to chat-window-integrated editors (like the first generation of Cursor or Windsurf), to today’s console agents, like Anthropic’s Claude Code or OpenAI’s Codex CLI. A console agent is quite a simple idea; it is essentially a loop around an LLM interaction with the ability to call external tools. A console agent’s tool calling model fits very neatly together with the “unix philosophy” of small, composable tools that do one thing. We now have a convenient way of extending the capabilities of AI-powered development tools and, of course, tools like CoPilot and Cursor are now fully “agentic” too.

Evaluating APL

With tool calling in an agent, we can give an LLM the ability to actually evaluate APL, which, in conjunction with the post-inflection point models, makes a real difference in the productivity you can get from LLMs when working with APL. There are several ways you can achieve this, the simplest being the dyalogscript CLI, which has shipped with Dyalog since Dyalog v19.0. Teaching Claude Code how to use dyalogscript is remarkably simple – just make Claude aware of its existence, and tell it to work out how to use it:

❯ Figure out how to use /usr/local/bin/dyalogscript to evaluate APL

[a few minutes of Claude poking dyalogscript]

●  Figured out. Here's how to evaluate APL with /usr/local/bin/dyalogscript.

What it is

dyalogscript (symlink → Dyalog 21.0's scriptbin/dyalogscript, a compiled binary) runs a file of APL source by launching dyalog ... -script FILE. It takes a script file argument — not an inline expression. (Your existing apl alias just opens the Dyalog 19.0 GUI app; dyalogscript is the headless evaluator.)

Basic usage

dyalogscript path/to/script.apls         # run a file
dyalogscript script.apls arg1 arg2       # args land in ⎕ARG
dyalogscript MAXWS=300Mb script.apls     # options go BEFORE the file (or on the shebang)
Inside a script: statements are separated by newlines or ⋄, output with ⎕←, and strings use single quotes ('hi', not "hi").

⚠️ The one real gotcha: output uses \r, not \n

⎕← terminates each line with a carriage return, so in a terminal/pipe lines overwrite each other and look concatenated (455Hello…). Fix by normalizing:
dyalogscript script.apls | tr '\r' '\n'

Ad-hoc evaluation without a temp file

All of these work because the wrapper only needs a readable file path:
# pipe via /dev/stdin (most portable)
printf '⎕←+/⍳10\n' | dyalogscript /dev/stdin | tr '\r' '\n'

# heredoc
dyalogscript /dev/stdin <<'APL' | tr '\r' '\n'
⎕←2+2
⎕←3 3⍴⍳9
APL

# process substitution
dyalogscript <(printf '⎕←⌽⍳5\n') | tr '\r' '\n'
Or make a script self-executable with a shebang (as the samples in …/Dyalog/Samples/scripts/ do):
#!/usr/local/bin/dyalogscript MAXWS=64Mb
⎕←'hello'

Errors

Uncaught errors print inline (error type, the offending line, and a ∧ caret) and halt the script. Note: exit code is always 0, even on error — so detect failures by scanning output, not $? (wrap risky code in :Trap/⎕TRAP if you need control flow).

Well done, Claude (despite the hallucination of ⎕ARG in a comment). However, doing this every time you start Claude soon gets tedious. Fortunately, Claude Code has a mechanism for encoding such things durably: skills. A “Claude skill” is a piece of Markdown prose that encodes some trigger condition with a brief description of the tool or behaviour that it’s concerned with. You can view/download my dyalogscript skill from the Dyalog Labs GitHub, or you could tell Claude to write one for you after it’s worked out how to use it.

Let’s ask Claude to pick up the skills file:

❯ Grab the skill from https://github.com/dyalog-labs/agent-dev-container/blob/main/.devcontainer/kit/.claude/skills/dyalog-script/SKILL.md

●  I'll grab the raw file. Let me fetch it from the raw GitHub URL.

●  Bash(curl -fsSL "https://... [snip]")

●  Got it — a self-contained skill. Let me check whether the skill directory has any sibling files (scripts/resources) I should grab too.

●  Bash(curl -fsSL "https://... [snip]")

●  Installed and verified — 1279 bytes, an exact match for GitHub's reported size, so it's byte-for-byte identical and the APL glyphs survived intact.

  Location: .claude/skills/dyalog-script/SKILL.md (it was the only file in that skill's directory — nothing else to grab).

  A few notes:

  - Availability: Skills load at startup, so dyalog-script will show up as an invocable skill in your next Claude Code session, not this one.
  - Scope: I installed it at project-local scope (/.claude/skills/), so it's available in this project. Say the word if you'd rather have it user-scope. (~/.claude/skills/).
  - Content: It matches exactly what we worked out last turn — dyalogscript /dev/stdin with echo/heredoc, and ⎕← to print. One thing the skill doesn't mention that we discovered: on this macOS install, ⎕← separates lines with \r, so for clean multi-line terminal output you may still want to pipe through tr '\r' '\n'.

We need to restart Claude Code for it to pick it up. After restart, hitting / shows the available skills and commands, and we should see our fresh dyalogscript skill:

Claude Code showing the available skills and commands

Claude Code showing the available skills and commands

Claude should now be able to run APL expressions by itself. Let’s test that (note that the apparently mangled lines are due to the line break issues mentioned above):

❯ Evaluate the following Dyalog APL expression and tell me how it works: 3 (⍴⍴1,0⍴⍨⊢+.*∘⍳-∘≢) 4

●  I'll analyze this expression carefully, then verify my reasoning by running it.

  Let me first work through the structure, then check it against the interpreter.

●  Skill(dyalog-script)
  ⎿  Successfully loaded skill

●  Bash(dyalogscript /dev/stdin <<'EAPL'
      ⎕←'=== full result ==='…)
  ⎿  [tool calling log removed for brevity]

●  Bash(dyalogscript /dev/stdin <<'EAPL'
      ⍝ break down the inner tines with X=3 (left), Y=4 (right)…)
  ⎿  [tool calling log removed for brevity] 

●  Confirmed. The result is a 4×4×4 identity tensor — a rank-3 array with 1s on the main diagonal (where all three indices are equal) and 0s everywhere else:

1 0 0 0      0 0 0 0      0 0 0 0      0 0 0 0
0 0 0 0      0 1 0 0      0 0 0 0      0 0 0 0
0 0 0 0      0 0 0 0      0 0 1 0      0 0 0 0
0 0 0 0      0 0 0 0      0 0 0 0      0 0 0 1

What it really is

This is a generalized identity-array generator. The left argument is the rank (3) and the right argument is the dimension (4), so 3 (…) 4 builds the 4×4×4 identity. (2 (…) 4 would give an ordinary 4×4 identity matrix.)

How it works

The parenthesised part is a train (tacit function). Operators bind tighter than the train, so first the derived functions form: +.*∘⍳ is (+.*)∘⍳, and -∘≢ is (-)∘(≢). That
leaves 9 tines, which—being odd-length—nest as right-grouped forks:

(⍴ ⍴ (1 , (0 ⍴⍨ (⊢ +.*∘⍳ -∘≢))))

Evaluating with X←3 (rank r), Y←4 (dimension n), inside-out:

Innermost fork (⊢ +.*∘⍳ -∘≢) — a fork applies as (X f Y) g (X h Y):
- Left tine 3⊢4 → 4 (the dimension n)
- Right tine 3(-∘≢)4 → 3 - ≢4 → 3 - 1 → 2 (this is r-1, the top exponent; ≢ of a scalar is 1)
- Middle 4 (+.*∘⍳) 2 → 4 +.* ⍳2 → 4 +.* 1 2 → 4^1 + 4^2 → 20

- That inner product computes Σ n^k for k=1…r-1 = 4 + 16 = the number of zeros needed.

0 ⍴⍨ 20 → 20 ⍴ 0 → twenty 0s.

1 , … → 1 0 0 … 0, length 21. This is the key: the vector's period is 1 + n + n²= 21.

Outer fork (⍴ ⍴ …):
- Left tine 3⍴4 → 4 4 4 (the shape: r copies of n)
- Reshape: 4 4 4 ⍴ (1,20⍴0)

Reshaping recycles the 21-element vector across 64 cells. The flat index of each diagonal element (i,i,i) is i × (1+n+n²) = i × 21, which lands exactly on the recycled 1s (flat indices 0, 21, 42, 63) — producing the identity tensor.

The idea in one line

It's the classic identity-matrix idiom n n ⍴ 1, n⍴0 lifted to arbitrary rank: the period (nʳ−1)/(n−1) = 1+n+…+nʳ⁻¹ makes a single 1 recur precisely on the main diagonal, and the train computes that period tacitly from the rank/dimension arguments.

✻ Crunched for 2m 24s

From this, we can note several things. Firstly, Claude worked out that it needed to load and invoke the dyalogscript skill, and secondly, it did a decent job of dissecting the fairly complex tacit expression for arbitrary rank identity matrix. It approached the job methodically, and tested as it progressed through the process.

At the time of writing, trying the same expression with the latest ChatGPT without the ability to evaluate returns a confident, but unfortunately nonsensical answer.

“Skills” is now an open standard, and OpenAI supports it, too. If I’d given OpenAI’s Codex tool the same skill, it would have solved it, too.

If you only take one thing from this blog post, then it should be this: enabling APL evaluation unlocks a lot of APL ability in the latest models.

Stateful APL evaluation

dyalogscript isn’t stateful – it runs an expression or file, and then exits. This doesn’t help you if you are working in a large, complex, APL application. For that you need a different mechanism, either connecting using the Ride port, or using an editor extension for something like VS Code. We’re currently working on such solutions, but that is out of scope for this blog post.

Containerising for Improved Safety

Safety when using AI is an important topic, and too big to do justice in a blog post like this. Running an agent locally, directly on your machine, exposes you to real risk: the agent can read, write, and delete files, install software, access credentials, access the local network and, of course, the web. Although the agents from reputable AI providers generally have both a good track record and plenty of internal guardrails, the risks are real. So what can you do if you want to experiment with AI agents whilst at the same time taking steps to minimise your exposure? One way is to run the agent in a container; this alone doesn’t mean “safe”, but it should at least decrease the blast radius.

I run Claude Code in a dev container, a Docker container that is configured to be seamlessly picked up by code editors like VS Code and Zed. We don’t publish a built container image, nor do we support it, but you can see and use its source code on Dyalog Labs, a GitHub organisation that we use at Dyalog Ltd specifically for experimenting with potentially-useful things that haven’t yet reached the standard for an “officially supported product”. The relevant repository is agent-dev-container – make sure that you examine it closely before you decide to make use of it.

Running Claude in the container means that it can see only the directory in which it was started and those below it. This container comes with an optional “starter kit” Claude code configuration and set-up – you already saw the dyalogscript skill. Let’s explore some of its features.

Start by cloning the dev container repository and lifting its configuration into our project repository:

~/work/tmp  $ git clone git@github.com:dyalog-labs/agent-dev-container.git
Cloning into 'agent-dev-container'...
remote: Enumerating objects: 67, done.
remote: Counting objects: 100% (67/67), done.
remote: Compressing objects: 100% (50/50), done.
remote: Total 67 (delta 21), reused 55 (delta 12), pack-reused 0 (from 0)
Receiving objects: 100% (67/67), 55.42 KiB | 2.52 MiB/s, done.
Resolving deltas: 100% (21/21), done.
~/work/tmp  $ mkdir my-project
~/work/tmp  $ cd my-project
~/work/tmp/my-project  $ cp -r ../agent-dev-container/.devcontainer .
~/work/tmp/my-project  $ code .

Opening the directory with VS Code lets it recognise that it contains a dev container and offer to open it in container mode. The first time we do this we trigger the container build, which can take several minutes to complete:

First open of dev container

First open of dev container

Open the terminal pane, and run the kit installation command to surface the Claude Code configuration (by keeping the dev container itself separate from the Claude configuration, you can choose to use either or both):

Install Claude "kit"

Install Claude “kit”

Now we can start Claude Code. As we’re in the container, we can enable the ominously named --dangerously-skip-permissions mode:

Start in yolo mode

Start in yolo mode

The first time that you need to log in to Claude, follow the instructions to land at an authenticated Claude session. If you have a Team subscription to Claude, Shift + Tab until you reach auto mode, which is the permissions mode that you want to be running in (keeping the bypass permissions mode for the rare occasions when you really need to escalate permissions).

Claude running

Claude running

This dev container comes equipped with development tool chains for Dyalog v20.0, C# .NET 8, Python 3.11, Go 1.24.1, and Node v20.20.2. It has the dyalogscript skill we showed earlier:

Claude running

Claude running

and the ability to read Dyalog’s documentation:

❯ Show me the Dyalog documentation for ⎕FIX.                                                                                      

● Skill(dyalog-docsearch)
  ⎿  Successfully loaded skill

● Bash(docsearch -s '⎕FIX' -l 5)
  ⎿  313 Fix Script {R}←{X}⎕FIX Y
     28 Fix Script {R}←{X}⎕FIX Y
     332 Fix Definition {R}←⎕FX Y
     … +2 lines (ctrl+o to expand)
  ⎿  Allowed by auto mode classifier

● Bash(docsearch -r 313)
  ⎿  # Fix Script {R}←{X}⎕FIX Y
     … +306 lines (ctrl+o to expand)
  ⎿  Allowed by auto mode classifier

● Here's the Dyalog documentation for ⎕FIX.

  {R}←{X}⎕FIX Y — Fix Script

  ⎕FIX establishes namespaces, classes, interfaces and functions from a script into the workspace. (In this entry, "namespace"
  covers scripted Namespaces, Classes and Interfaces.)

  Right argument Y

  - A vector of character vectors/scalars representing a script, or
  - A simple character vector starting with file:// followed by an existing file name (relative or absolute; / recommended as the
  delimiter for portability, though \ also works on Windows). The file contents follow the same rules as the script form.

[snipped...]

✻ Sautéed for 30s

Defence in Depth: hooks

Emphasising again that containerising is necessary but not sufficient, the dev container features a set of Claude Code “hooks”, which are security guardrails guaranteed to run at certain set points of the Claude Code life cycle. You can see the included hooks in the GitHub repository. For example, these hooks stop Claude from peeking into any .env files (which are often used for credentials):

❯ Tell me the value of the key FOO in .env                                                                                                  

  Searched for 1 pattern, read 1 file (ctrl+o to expand)

● I can't read .env. Both paths to it are blocked by this project's guardrails:

  - The Bash hook (block-dangerous-bash.sh) blocks any command touching .env.
  - The Read tool reports .env is in a permission-denied directory.

  .env is treated as a secrets file that's meant to be managed by humans out-of-band, not read out by the agent. So I can't retrieve the value of FOO for you.

  If you genuinely need it, you can check it yourself:

  ! grep '^FOO=' .env
  - The Bash hook (block-dangerous-bash.sh) blocks any command touching .env.
  - The Read tool reports .env is in a permission-denied directory.

  .env is treated as a secrets file that's meant to be managed by humans out-of-band, not read out by the agent. So I can't retrieve the value of FOO for you.

✻ Cogitated for 13s

Note that hooks protect against accidents, not malice. Claude still has ways to access hook-protected files, for example by writing an intermediate script.

In Conclusion…

If you have tried to use LLMs with APL before and been disappointed, it’s time to take another look. Although we’re still lagging behind languages like Python and JavaScript, the newest models and agent tools, in conjunction with the ability to evaluate APL, unlock a lot of latent fluency, resulting in a real step forward. Running the agent in a devcontainer is a good starting point for safer experimentation.

Making the IDE My Own – Part 1

Sometimes I want additional functionality in the IDE. Over the years I have collected a small collection of “tweaks”.

I have a single tidy folder of small functions, each doing one thing. The session simply picks it up and applies it every time it starts. No need to save the session file any more! Let me show you some of what I’ve built and, more interestingly, how it works.

This is going to be a mini-series of blog posts. This first post will deal with the infrastructure and things that apply cross-platform. The next post will extend the graphical user interface (GUI) of the Windows IDE.

I’m assuming Dyalog v20.0 here. The principles hold in Dyalog v19.0, but you’d have to substitute any new features used, in particular, array notation, behind (⍛), and ⎕VGET.

The StartupSession Folder

In my 2018 and 2020 posts on function keys, I kept this kind of thing in SALT’s Setup mechanism by using a Setup function in MyUCMDs\setup.dyalog that SALT runs at startup. That still works, but I have since migrated to the new session initialisation, which is altogether more versatile; it doesn’t only run code, it can also leave functions (and more) resident in ⎕SE afterwards.

The way it works is that, at startup, Dyalog populates the session namespace (⎕SE) from StartupSession folders within your documents folder, and, once everything has loaded, it automatically calls the Run function in each sub-folder of a StartupSession folder. There is one StartupSession folder pertaining to all versions of Dyalog plus a separate one for each of the versions installed.

For example, under Microsoft Windows, I’d place all the functions listed in this blog post into a folder I’ve named C:\Users\adam\Documents\Dyalog APL Files\StartupSession\seext (for session extensions). If I wanted it just for a specific version, such as Dyalog v20.0, I could put it into C:\Users\adam\Documents\Dyalog APL-64 20.0 Unicode Files\StartupSession\seext instead. If you also have older versions installed you might want to do exactly that, since the code as listed requires Dyalog v20.0 or later.

On other platforms, the corresponding folders would be /home/adam/dyalog.files/StartupSession/seext and /home/adam/dyalog.200U64.files/StartupSession/seext, for all versions and specific versions respectively.

The functions live in ⎕SE, so they are available even if the workspace is cleared or another one is loaded. They also do not interfere with the workspace content. I keep two types of functions side by side:

Feature
Begins with a lowercase letter. Enables a specific behaviour.
Utility
Begins with an uppercase letter. Supports the functionality of a feature.

In this post I’ll only be dealing with the first type, but I’ll prepare the special Run function (technically a utility) so it is ready for next time. Run is the bootstrapping function that calls the feature functions:

∇ Run args
 ⍝ Call niladic fns with lowercase initial except mentioned in exclude.config
  ;path;exclude
  :If ⎕NEXISTS path←args⊃⍛,'/exclude.config'
      exclude←⎕NGET path 1
  :Else
      exclude←0⍴⊂''
  :EndIf
  ⍎¨exclude~⍨{0=11 ⎕ATX ⍵}⍛/⎕A ⎕C⍛⎕NL ¯3
∇

The session initialisation code hands it a vector argument in which the first element specifies the directory from where it was loaded. The exclude-list’s filename is appended, and, if it exists, the exclude-list is read in as a vector of character vectors; otherwise, we just set a null list. Then comes a somewhat involved line, that benefits from being broken into chunks – reading these from right to left:

  1. ⎕A ⎕C⍛⎕NL ¯3 asks for the names. ⎕C is Case Convert. The behind operator ⍛ modifies ⎕NL so that it pre-processes its left argument to be lowercased (actually case-folded, but it doesn’t matter for the basic Latin alphabet). So, we are really evaluating 'abcdefghijklmnopqrstuvwxyz' ⎕NL ¯3, that is, “give me the names of all functions (3) beginning with a lowercase Latin letter” as a nested vector of names.
  2. {0=11 ⎕ATX ⍵}⍛/ then filters that list. ⎕ATX reports extended attributes of a name; attribute 11 is the function’s valence, so 0= keeps only the niladic ones. The ⍛/ construct is my precious derived monadic filtering operator: Predicate⍛/names is Predicate filtering names, or (Predicate names)/names pre-behind.
  3. exclude~⍨ removes the features that have been manually excluded, if any.
  4. ⍎¨ calls each survivor of the culling.

The lowercase requirement isn’t arbitrary. In accordance with my personal naming convention, since a niladic function has the syntactic role of an array, its name needs a lowercase initial. This means that either test (lowercase initial or niladic valence) would already pick out exactly the features. Run checks both anyway; I like to err on the side of caution. Adding a feature or utility then simply involves dropping it into the folder; there is nothing to register, and as long as I adhere to my naming convention, it all just works. If you want to install the features, but want some disabled, create an exclude.config file with one feature function name on each line.

Now to the features themselves.

Output Settings

Let’s begin by tidying up appearances. setOutput configures how results are displayed:

∇ setOutput
 ⍝ Set ]Box and ]Rows
  ⎕SE.UCMD¨(
      '←OUTPUT.Box on -f=on -t=tree'
      '←OUTPUT.Rows on -s=long -fold=3'
  )
∇

These are the ]Box and ]Rows user commands. A leading ← on a user command invocation silences it, so the session comes up without two lines of Was OFF confirmations. The modifiers are where the behaviour lives (note that you can abbreviate any modifier as long as it stays unambiguous):

]Box
-t=tree
Short for -trains=tree, draws tacit functions as trees rather than boxes.
-f=on
Short for -fns=on, makes all of this apply to output produced inside functions (implicitly or using ⎕←), in addition to results and outputs that were requested at the Session level.
]Rows
-s=long
Short for -style=long, lets the session scroll horizontally instead of hard-wrapping long lines
-fold=3
Prevents a single tall result from flooding the screen; when output won’t fit vertically, the middle lines are replaced with leader dots and only the last 3 rows are shown.

Traditionally, you’d save your session file to preserve these output settings, but since our code runs at every startup, there’s no need for that.

One Log for Each Interpreter

109⌶ controls the file to which Dyalog writes its log of deprecated-feature usage. I run several interpreters – combinations of different versions, Unicode and Classic editions, 32- and 64-bit widths – and I don’t want them all writing to one file, so setLog gives each its own, named after the active interpreter and parked in the temporary directory:

∇ setLog
 ⍝ Set log file for usage of deprecated features
  ;tmp;Log;file;target;version;platform;type
  tmp←739⌶0
  Log←109⌶
  file←tmp,'/'
  (target version platform type)←# ⎕WG'APLVersion'
  file,←  3↑version∩⎕D
  file,←⊃'CU'⌽⍨80=⎕DR''
  file,←¯2↑'32',target∩⎕D
  file,←'.log'
  file Log 0
∇

(I rather enjoy the mnemonics of the two I-beams: 109 reads as the letters log, and 739 as TMP — a slanted T, a sideways M, a mirrored P). From APLVersion we keep the first three version digits, then choose 'U' or 'C' (80=⎕DR'' asks whether the empty character vector has a Data Representation of 80, meaning 8-bit Unicode, as opposed to 82 for 8-bit Classic, and 'CU'⌽⍨ rotates the pair so the correct letter falls first), then append '64' or '32' from the target platform. Under a Unicode-edition 64-bit Dyalog v20.0, this gives 200U64.log.

A Name and a Home

When I open a fresh interpreter (I do this a lot, and often have multiple running at the same time) to try something out, I often forget to save my noodling before I close the interpreter (or it crashes…). In addition, due to old habits, I might end up saving a workspace rather than using Link. autoLink solves these issues:

∇ {msg}←autoLink
 ⍝ If CLEAR WS, Link # to timestamped dir and set ⎕WSID
  ;tmp;∆DT;path
  tmp←739⌶0
  ∆DT←1200⌶
  path←⎕SE.Dyalog.Utils.Config'LOAD'
  :If ''≡path
  :AndIf 'CLEAR WS'≡⎕WSID
  :AndIf (⊂⍕#)~⍤∊(⎕VGET⊂'⎕SE.Link.Links'(ns:0)).ns
      path←tmp,⊃'/YYYYMMDD.hhmmss'∆DT 1 ⎕DT'J'
      ⎕←msg←⎕SE.Link.Create # path
      ⎕WSID←path,'/'
  :EndIf
∇

This function does three things:

  1. First, it checks whether there is anything already loaded in or linked to the root namespace (#).
  2. Then, if there is nothing, it creates a link between # and a new directory named with the current local timestamp.
  3. Finally, it sets the Workspace Identification (⎕WSID) to the path of the newly-created linked directory.

As you can see, I lean heavily on Dyalog v20.0 features here: ⎕VGET reads ⎕SE.Link.Links (Link’s memory of active links), but supplies the array-notation namespace (ns:0) as a default, so the line can’t fall over with a VALUE ERROR (if the list hasn’t even been created) or a NONCE ERROR (because there are no currently active links).

Now, every function I create or modify using the Editor is written straight to disk as source (a throwaway scratchpad that nonetheless survives an early shut-down or a crash).

Setting ⎕WSID has two positive effects:

  • Both Ride and the Windows IDE display ⎕WSID in the titlebar, this helps me with situational awareness when I have many interpreter instances running in parallel.
  • The trailing slash in ⎕WSID makes it an invalid filename, preventing me from accidentally saving the workspace (which would create a workspace with the same name as the directory, but with the file extension **.dws**, and with active links, both of which could lead to all sorts of unpleasantness), and further increases my situational awareness by telling me that the current workspace has a directory as source, rather than a binary blob workspace file.

To Be Continued…

So far, I have looked at the setup, making output readable, and ensuring code is saved. In my next post I’ll look at customising the GUI. In the meantime, the easiest way to try out anything from this post is to paste all the function definitions into a clear workspace and enter (with the directory that’s appropriate for you):

]LINK.Export # C:\Users\adam\Documents\Dyalog APL-64 20.0 Unicode Files\StartupSession\seext

If any of this is useful to you, take it. If it doesn’t quite do what you want, adapt it. If you have an idea for additional cross-platform Session extensions, let me know!