Working with LLMs and Dyalog

Notes about this blog post:

  • In this blog post, I’m using macOS and Docker. If you’re a Microsoft Windows user and you want to follow along, Docker works well on WSL2.
  • AI is a fast-moving field, and I’ll assume some familiarity with LLMs in terms of terminology.

Working with an AI coding “agent” can make us more productive by automating boiler plating, helping with testing, and so on. LLMs are very good at languages like Python and C#, but have struggled with APL. We can speculate as to why this is, but the answer is most likely as mundane as the lack of APL “out there”. The training sets are sparse when it comes to APL. The other aspect is one of incentives: no frontier AI laboratory has any real incentive to make their models better at APL; they’re usually Python shops.

However, the pace of improvement in LLMs generally, and the drive towards “smarter, not bigger” models, now make LLMs viable as a productivity tool when working with Dyalog APL. The latest models from Anthropic, OpenAI, and Google today write passable APL – still a bit naive and Python-like, but capable of writing functional, non-trivial APL. In this blog post, I will outline my working practices and set-up, and a few practical tips on how to make an LLM more fluent in APL.

Warning: using an AI coding agent requires care. Although running an AI agent in a container narrows the blast radius, many risks remain unavoidable, especially when running in more autonomous modes.

“The November 2025 Inflection Point”

Two model releases happened in November, 2025 – Claude Opus 4.5 and GPT-5.2 – within days of each other. Simon Willison coined the expression The November 2025 Inflection Point for this quantum leap. Up until then, LLM performance on APL was abysmal: they were unable to understand the right-to-left execution order and really struggled with the array model in general, let alone syntax. After the November release, all of that changed. Although the models were still not exactly fluent in APL, it was a real step change, especially in their ability to explain APL code. APL performance is lifted by the general improvement in models over time – it remains far behind the performance in more mainstream languages, but is now something that is approaching useful to APL programmers.

Tooling Improvements

In conjunction with the model improvements, suddenly “Agents” took off. How we interact with LLMs has also evolved, from the original CoPilot smart auto-complete, to copy-pasting code into the ChatGPT web interface, to chat-window-integrated editors (like the first generation of Cursor or Windsurf), to today’s console agents, like Anthropic’s Claude Code or OpenAI’s Codex CLI. A console agent is quite a simple idea; it is essentially a loop around an LLM interaction with the ability to call external tools. A console agent’s tool calling model fits very neatly together with the “unix philosophy” of small, composable tools that do one thing. We now have a convenient way of extending the capabilities of AI-powered development tools and, of course, tools like CoPilot and Cursor are now fully “agentic” too.

Evaluating APL

With tool calling in an agent, we can give an LLM the ability to actually evaluate APL, which, in conjunction with the post-inflection point models, makes a real difference in the productivity you can get from LLMs when working with APL. There are several ways you can achieve this, the simplest being the dyalogscript CLI, which has shipped with Dyalog since Dyalog v19.0. Teaching Claude Code how to use dyalogscript is remarkably simple – just make Claude aware of its existence, and tell it to work out how to use it:

❯ Figure out how to use /usr/local/bin/dyalogscript to evaluate APL

[a few minutes of Claude poking dyalogscript]

●  Figured out. Here's how to evaluate APL with /usr/local/bin/dyalogscript.

What it is

dyalogscript (symlink → Dyalog 21.0's scriptbin/dyalogscript, a compiled binary) runs a file of APL source by launching dyalog ... -script FILE. It takes a script file argument — not an inline expression. (Your existing apl alias just opens the Dyalog 19.0 GUI app; dyalogscript is the headless evaluator.)

Basic usage

dyalogscript path/to/script.apls         # run a file
dyalogscript script.apls arg1 arg2       # args land in ⎕ARG
dyalogscript MAXWS=300Mb script.apls     # options go BEFORE the file (or on the shebang)
Inside a script: statements are separated by newlines or ⋄, output with ⎕←, and strings use single quotes ('hi', not "hi").

⚠️ The one real gotcha: output uses \r, not \n

⎕← terminates each line with a carriage return, so in a terminal/pipe lines overwrite each other and look concatenated (455Hello…). Fix by normalizing:
dyalogscript script.apls | tr '\r' '\n'

Ad-hoc evaluation without a temp file

All of these work because the wrapper only needs a readable file path:
# pipe via /dev/stdin (most portable)
printf '⎕←+/⍳10\n' | dyalogscript /dev/stdin | tr '\r' '\n'

# heredoc
dyalogscript /dev/stdin <<'APL' | tr '\r' '\n'
⎕←2+2
⎕←3 3⍴⍳9
APL

# process substitution
dyalogscript <(printf '⎕←⌽⍳5\n') | tr '\r' '\n'
Or make a script self-executable with a shebang (as the samples in …/Dyalog/Samples/scripts/ do):
#!/usr/local/bin/dyalogscript MAXWS=64Mb
⎕←'hello'

Errors

Uncaught errors print inline (error type, the offending line, and a ∧ caret) and halt the script. Note: exit code is always 0, even on error — so detect failures by scanning output, not $? (wrap risky code in :Trap/⎕TRAP if you need control flow).

Well done, Claude (despite the hallucination of ⎕ARG in a comment). However, doing this every time you start Claude soon gets tedious. Fortunately, Claude Code has a mechanism for encoding such things durably: skills. A “Claude skill” is a piece of Markdown prose that encodes some trigger condition with a brief description of the tool or behaviour that it’s concerned with. You can view/download my dyalogscript skill from the Dyalog Labs GitHub, or you could tell Claude to write one for you after it’s worked out how to use it.

Let’s ask Claude to pick up the skills file:

❯ Grab the skill from https://github.com/dyalog-labs/agent-dev-container/blob/main/.devcontainer/kit/.claude/skills/dyalog-script/SKILL.md

●  I'll grab the raw file. Let me fetch it from the raw GitHub URL.

●  Bash(curl -fsSL "https://... [snip]")

●  Got it — a self-contained skill. Let me check whether the skill directory has any sibling files (scripts/resources) I should grab too.

●  Bash(curl -fsSL "https://... [snip]")

●  Installed and verified — 1279 bytes, an exact match for GitHub's reported size, so it's byte-for-byte identical and the APL glyphs survived intact.

  Location: .claude/skills/dyalog-script/SKILL.md (it was the only file in that skill's directory — nothing else to grab).

  A few notes:

  - Availability: Skills load at startup, so dyalog-script will show up as an invocable skill in your next Claude Code session, not this one.
  - Scope: I installed it at project-local scope (/.claude/skills/), so it's available in this project. Say the word if you'd rather have it user-scope. (~/.claude/skills/).
  - Content: It matches exactly what we worked out last turn — dyalogscript /dev/stdin with echo/heredoc, and ⎕← to print. One thing the skill doesn't mention that we discovered: on this macOS install, ⎕← separates lines with \r, so for clean multi-line terminal output you may still want to pipe through tr '\r' '\n'.

We need to restart Claude Code for it to pick it up. After restart, hitting / shows the available skills and commands, and we should see our fresh dyalogscript skill:

Claude Code showing the available skills and commands

Claude Code showing the available skills and commands

Claude should now be able to run APL expressions by itself. Let’s test that (note that the apparently mangled lines are due to the line break issues mentioned above):

❯ Evaluate the following Dyalog APL expression and tell me how it works: 3 (⍴⍴1,0⍴⍨⊢+.*∘⍳-∘≢) 4

●  I'll analyze this expression carefully, then verify my reasoning by running it.

  Let me first work through the structure, then check it against the interpreter.

●  Skill(dyalog-script)
  ⎿  Successfully loaded skill

●  Bash(dyalogscript /dev/stdin <<'EAPL'
      ⎕←'=== full result ==='…)
  ⎿  [tool calling log removed for brevity]

●  Bash(dyalogscript /dev/stdin <<'EAPL'
      ⍝ break down the inner tines with X=3 (left), Y=4 (right)…)
  ⎿  [tool calling log removed for brevity] 

●  Confirmed. The result is a 4×4×4 identity tensor — a rank-3 array with 1s on the main diagonal (where all three indices are equal) and 0s everywhere else:

1 0 0 0      0 0 0 0      0 0 0 0      0 0 0 0
0 0 0 0      0 1 0 0      0 0 0 0      0 0 0 0
0 0 0 0      0 0 0 0      0 0 1 0      0 0 0 0
0 0 0 0      0 0 0 0      0 0 0 0      0 0 0 1

What it really is

This is a generalized identity-array generator. The left argument is the rank (3) and the right argument is the dimension (4), so 3 (…) 4 builds the 4×4×4 identity. (2 (…) 4 would give an ordinary 4×4 identity matrix.)

How it works

The parenthesised part is a train (tacit function). Operators bind tighter than the train, so first the derived functions form: +.*∘⍳ is (+.*)∘⍳, and -∘≢ is (-)∘(≢). That
leaves 9 tines, which—being odd-length—nest as right-grouped forks:

(⍴ ⍴ (1 , (0 ⍴⍨ (⊢ +.*∘⍳ -∘≢))))

Evaluating with X←3 (rank r), Y←4 (dimension n), inside-out:

Innermost fork (⊢ +.*∘⍳ -∘≢) — a fork applies as (X f Y) g (X h Y):
- Left tine 3⊢4 → 4 (the dimension n)
- Right tine 3(-∘≢)4 → 3 - ≢4 → 3 - 1 → 2 (this is r-1, the top exponent; ≢ of a scalar is 1)
- Middle 4 (+.*∘⍳) 2 → 4 +.* ⍳2 → 4 +.* 1 2 → 4^1 + 4^2 → 20

- That inner product computes Σ n^k for k=1…r-1 = 4 + 16 = the number of zeros needed.

0 ⍴⍨ 20 → 20 ⍴ 0 → twenty 0s.

1 , … → 1 0 0 … 0, length 21. This is the key: the vector's period is 1 + n + n²= 21.

Outer fork (⍴ ⍴ …):
- Left tine 3⍴4 → 4 4 4 (the shape: r copies of n)
- Reshape: 4 4 4 ⍴ (1,20⍴0)

Reshaping recycles the 21-element vector across 64 cells. The flat index of each diagonal element (i,i,i) is i × (1+n+n²) = i × 21, which lands exactly on the recycled 1s (flat indices 0, 21, 42, 63) — producing the identity tensor.

The idea in one line

It's the classic identity-matrix idiom n n ⍴ 1, n⍴0 lifted to arbitrary rank: the period (nʳ−1)/(n−1) = 1+n+…+nʳ⁻¹ makes a single 1 recur precisely on the main diagonal, and the train computes that period tacitly from the rank/dimension arguments.

✻ Crunched for 2m 24s

From this, we can note several things. Firstly, Claude worked out that it needed to load and invoke the dyalogscript skill, and secondly, it did a decent job of dissecting the fairly complex tacit expression for arbitrary rank identity matrix. It approached the job methodically, and tested as it progressed through the process.

At the time of writing, trying the same expression with the latest ChatGPT without the ability to evaluate returns a confident, but unfortunately nonsensical answer.

“Skills” is now an open standard, and OpenAI supports it, too. If I’d given OpenAI’s Codex tool the same skill, it would have solved it, too.

If you only take one thing from this blog post, then it should be this: enabling APL evaluation unlocks a lot of APL ability in the latest models.

Stateful APL evaluation

dyalogscript isn’t stateful – it runs an expression or file, and then exits. This doesn’t help you if you are working in a large, complex, APL application. For that you need a different mechanism, either connecting using the Ride port, or using an editor extension for something like VS Code. We’re currently working on such solutions, but that is out of scope for this blog post.

Containerising for Improved Safety

Safety when using AI is an important topic, and too big to do justice in a blog post like this. Running an agent locally, directly on your machine, exposes you to real risk: the agent can read, write, and delete files, install software, access credentials, access the local network and, of course, the web. Although the agents from reputable AI providers generally have both a good track record and plenty of internal guardrails, the risks are real. So what can you do if you want to experiment with AI agents whilst at the same time taking steps to minimise your exposure? One way is to run the agent in a container; this alone doesn’t mean “safe”, but it should at least decrease the blast radius.

I run Claude Code in a dev container, a Docker container that is configured to be seamlessly picked up by code editors like VS Code and Zed. We don’t publish a built container image, nor do we support it, but you can see and use its source code on Dyalog Labs, a GitHub organisation that we use at Dyalog Ltd specifically for experimenting with potentially-useful things that haven’t yet reached the standard for an “officially supported product”. The relevant repository is agent-dev-container – make sure that you examine it closely before you decide to make use of it.

Running Claude in the container means that it can see only the directory in which it was started and those below it. This container comes with an optional “starter kit” Claude code configuration and set-up – you already saw the dyalogscript skill. Let’s explore some of its features.

Start by cloning the dev container repository and lifting its configuration into our project repository:

~/work/tmp  $ git clone git@github.com:dyalog-labs/agent-dev-container.git
Cloning into 'agent-dev-container'...
remote: Enumerating objects: 67, done.
remote: Counting objects: 100% (67/67), done.
remote: Compressing objects: 100% (50/50), done.
remote: Total 67 (delta 21), reused 55 (delta 12), pack-reused 0 (from 0)
Receiving objects: 100% (67/67), 55.42 KiB | 2.52 MiB/s, done.
Resolving deltas: 100% (21/21), done.
~/work/tmp  $ mkdir my-project
~/work/tmp  $ cd my-project
~/work/tmp/my-project  $ cp -r ../agent-dev-container/.devcontainer .
~/work/tmp/my-project  $ code .

Opening the directory with VS Code lets it recognise that it contains a dev container and offer to open it in container mode. The first time we do this we trigger the container build, which can take several minutes to complete:

First open of dev container

First open of dev container

Open the terminal pane, and run the kit installation command to surface the Claude Code configuration (by keeping the dev container itself separate from the Claude configuration, you can choose to use either or both):

Install Claude "kit"

Install Claude “kit”

Now we can start Claude Code. As we’re in the container, we can enable the ominously named --dangerously-skip-permissions mode:

Start in yolo mode

Start in yolo mode

The first time that you need to log in to Claude, follow the instructions to land at an authenticated Claude session. If you have a Team subscription to Claude, Shift + Tab until you reach auto mode, which is the permissions mode that you want to be running in (keeping the bypass permissions mode for the rare occasions when you really need to escalate permissions).

Claude running

Claude running

This dev container comes equipped with development tool chains for Dyalog v20.0, C# .NET 8, Python 3.11, Go 1.24.1, and Node v20.20.2. It has the dyalogscript skill we showed earlier:

Claude running

Claude running

and the ability to read Dyalog’s documentation:

❯ Show me the Dyalog documentation for ⎕FIX.                                                                                      

● Skill(dyalog-docsearch)
  ⎿  Successfully loaded skill

● Bash(docsearch -s '⎕FIX' -l 5)
  ⎿  313 Fix Script {R}←{X}⎕FIX Y
     28 Fix Script {R}←{X}⎕FIX Y
     332 Fix Definition {R}←⎕FX Y
     … +2 lines (ctrl+o to expand)
  ⎿  Allowed by auto mode classifier

● Bash(docsearch -r 313)
  ⎿  # Fix Script {R}←{X}⎕FIX Y
     … +306 lines (ctrl+o to expand)
  ⎿  Allowed by auto mode classifier

● Here's the Dyalog documentation for ⎕FIX.

  {R}←{X}⎕FIX Y — Fix Script

  ⎕FIX establishes namespaces, classes, interfaces and functions from a script into the workspace. (In this entry, "namespace"
  covers scripted Namespaces, Classes and Interfaces.)

  Right argument Y

  - A vector of character vectors/scalars representing a script, or
  - A simple character vector starting with file:// followed by an existing file name (relative or absolute; / recommended as the
  delimiter for portability, though \ also works on Windows). The file contents follow the same rules as the script form.

[snipped...]

✻ Sautéed for 30s

Defence in Depth: hooks

Emphasising again that containerising is necessary but not sufficient, the dev container features a set of Claude Code “hooks”, which are security guardrails guaranteed to run at certain set points of the Claude Code life cycle. You can see the included hooks in the GitHub repository. For example, these hooks stop Claude from peeking into any .env files (which are often used for credentials):

❯ Tell me the value of the key FOO in .env                                                                                                  

  Searched for 1 pattern, read 1 file (ctrl+o to expand)

● I can't read .env. Both paths to it are blocked by this project's guardrails:

  - The Bash hook (block-dangerous-bash.sh) blocks any command touching .env.
  - The Read tool reports .env is in a permission-denied directory.

  .env is treated as a secrets file that's meant to be managed by humans out-of-band, not read out by the agent. So I can't retrieve the value of FOO for you.

  If you genuinely need it, you can check it yourself:

  ! grep '^FOO=' .env
  - The Bash hook (block-dangerous-bash.sh) blocks any command touching .env.
  - The Read tool reports .env is in a permission-denied directory.

  .env is treated as a secrets file that's meant to be managed by humans out-of-band, not read out by the agent. So I can't retrieve the value of FOO for you.

✻ Cogitated for 13s

Note that hooks protect against accidents, not malice. Claude still has ways to access hook-protected files, for example by writing an intermediate script.

In Conclusion…

If you have tried to use LLMs with APL before and been disappointed, it’s time to take another look. Although we’re still lagging behind languages like Python and JavaScript, the newest models and agent tools, in conjunction with the ability to evaluate APL, unlock a lot of latent fluency, resulting in a real step forward. Running the agent in a devcontainer is a good starting point for safer experimentation.

Maintaining Py’n’APL Part 2: APL Arrays, Python Objects, and JSON

As part of the bigger, overarching refactoring goal of making Py’n’APL great again, I refactored some of the code that deals with sending data from Python to APL and receiving data from APL into Python. In this blog post, I will describe – to the best of my abilities – how that part of the code works, how it differs from what was in place, and why those changes were made.

The starting point for this blog post is the commit b7d4749.

This blog post is mostly concerned with the files Array.py, ConversionInterface.py, and ObjectWrapper.py (these were the original file names before I tore them apart and moved things around). It does not make much sense to list where all the things went, but you can use GitHub’s compare feature to compare the starting commit for this blog post with the “final” commit for this blog post.

State of Affairs

If you are going to refactor a working piece of code, the first thing you need to do is to make sure that you know what the code is doing! This will help to ensure that your refactoring does not break the functionality of the code. With that in mind, I started working my way through the code.

I started by looking at the file ConversionInterface.py and the two classes Sendable and Receivable that were defined in there. By reading the comments, I understood that these two classes were defining the “conversion interface”. In this context, the word “interface” has approximately the Java meaning of interface: it defines a set of methods that the classes that inherit from these base classes have to implement. For the class Sendable, there are two methods toJSONDict and toJSONString; and for the class Receivable, there is one method to_python.

Even though I had just started, I already had a couple of questions:

  1. Do the names Sendable and Receivable mean that these objects will be sent to/received from APL or from Python respectively?
  2. Why is there a comment next to the definition of Sendable that says that classes that implement a method from_python will inherit from Sendable? Is that a comment that became a lie as the code evolved? If not, why isn’t there a stub for that method in the class itself?

The more I pondered on these questions, the more I started to think that the “conversion interface” isn’t necessarily about the sending to/receiving from APL, but rather the conversion of built-in Python types to helper classes like APLArray or APLNamespace (from the file Array.py) and back. So, it might be that Sendable and Receivable are supposed to be base classes for these helper classes, telling us which ones can be converted to/from built-in Python types. I needed to solve this conundrum before I could prepare these two base classes and use Python mechanisms to enforce these “interfaces”.

What the Interface Really Means

After playing around with the code a bit more, I felt more confident that Sendable should be inherited by classes that represent things that can be sent to APL and Receivable represents things that can be received from APL. However, it must be noted that Py’n’APL doesn’t send Python built-in types directly to APL. Whenever we want to send something to APL, Py’n’APL first converts it to the suitable intermediate (Python) class. For example, lists and tuples are converted to APLArray, and dictionaries are converted to APLNamespace.

If an APLArray instance is supposed to be sendable to APL, we must first be able to build it from the corresponding Python built-in types, and that is why almost all Sendable subclasses also implement a method from_python. Looking at it from the other end of the connection, Receivable instances come from APL and Py’n’APL starts by taking the JSON and converting it into the appropriate APLArray instances, APLNamespace instances, etc. Only then can we convert those intermediate representations to Python, and that is why all Receivable subclasses come with a method to_python. In addition, those Receivable instances come from APL as JSON, so we need to be able to instantiate them from JSON. That is why Receivable subclasses also implement a method fromJSONString, although that is not defined in the Receivable interface.

So, we have established that APL needs to know how to make sense of Python’s objects and Python needs to know how to make sense of APL’s arrays. (In Python, everything is an object, and in APL, everything is an array. In less precise – but maybe clearer – words, Python needs to be able to handle whatever APL passes to it, and APL needs to be able to handle whatever Python passes to it.) To implement this, we need to determine how Python objects map to APL arrays and how APL arrays map to Python objects. This is not trivial, otherwise I wouldn’t be writing about it! Here are two simple examples showing why this is not trivial:

  • Python does not have native support for arrays of arbitrary rank.
  • APL does not have a key-value mapping type like Python’s dict.

To solve the issues around Python and APL not having exactly the same type of data, we create lossless intermediate representations in both host languages. For example, Python needs to have an intermediate representation for APL arrays so that we can preserve rank information in Python. When possible, intermediate representations should know how to convert into the closest value in the host language. For example, the Python intermediate representation of a high-rank APL array should know how to convert itself into a Python list.

I began by looking at the handling of APL arrays and namespaces. These are the conversions that need to be in place:

  • APL arrays ←→ Python lists
  • APL arrays ← arbitrary Python iterables
  • APL namespaces ←→ Python dictionaries

When sending data from the Python side, it first needs to be converted into an instance of the appropriate APLProxy subclass. For example, a dictionary will be converted into an instance of APLNamespace. That object is converted to JSON, which is then sent to APL. APL receives the JSON and looks for a special field __extended_json_type__, which identifies the type of object. In this example, that is "APLNamespace". APL then uses that information to decode the JSON data into the appropriate thing (a namespace in this example).

When sending data from the APL side, a similar thing happens. First, the object is converted into a namespace that ⎕JSON knows how to handle. For example, an array becomes a namespace with attributes shape (the shape of the original array) and data (the ravel of the original array); the namespace is tagged with an attribute __extended_json_type__, which is a simple character vector informing Python what the object is. That namespace gets converted to JSON with ⎕JSON, and the JSON is sent to Python. Python receives the JSON and decodes it into a Python dictionary. Python then uses __extended_json_type__ to determine the actual object that the dictionary represents (an array, in our example) and uses the information available to build an instance of the appropriate APLProxy subclass (APLArray in this example).

Github commit 40523b9 shows one initial implementation of the APL code that takes APL arrays and namespaces and converts them into namespaces that ⎕JSON can handle and that Python knows how to interpret. This commit also shows the APL code for the reverse operation. For now, this APL code lives in the file Proxies.apln and the respective Python code lives in the file proxies.py. Everything is ready for me to hook this into the Py’n’APL machinery so that Py’n’APL uses this mechanism to pass data around…but that’s for another blog post!

Summary of Changes

GitHub’s compare feature shows all the changes I made since the commit that was the starting point for this post. The most notable changes are:

  • Moving the contents of ConversionInterface.py and ObjectWrapper.py into Array.py.
  • Adding the file proxies.py that will have the Python code to deal with the JSON and conversions, which will end up replacing most of the code I mentioned in the previous bullet point.
  • Adding the file Proxies.apln that will have the APL code to deal with the JSON and conversions, which will end up replacing a chunk of code that currently lives in Py.dyalog, which is a huge file with almost all of the Py’n’APL APL code.

Blog posts in this series:

Maintaining Py’n’APL Part 1: The Beginning

Py’n’APL is an interface between APL and Python that allows you to run Python code from within APL and APL code from within Python. This interface was originally developed by Dyalog Ltd intern Marinus Oosters, who presented it in a webinar and at Dyalog ’17. I subsequently talked about Py’n’APL at Dyalog ’21, where I promised to update it and make it into an awesome and robust tool.

I’ve now stared at Py’n’APL’s code base for longer than I’m proud to admit, but without any proper goals and some basic project management this has been as effective in cleaning it up as a Magikarp’s Splash – in other words, it has had no effect.

For that reason, and in another attempt to take up the maintenance of Py’n’APL, I have decided to start blogging about my progress. This will be a way for me to share with the world what it feels like to take up the maintenance of a project that you aren’t necessarily very familiar with.

(By the way, Py’n’APL is open source and has a very permissive licence. This means that, like me, you can also stare at the source code; it also means that you can go to GitHub, star the project, fork it, and play around with it!)

Tasks

There are some obvious tasks that I need to do, like testing Py’n’APL thoroughly. This will help make Py’n’APL more robust, it will certainly uncover bugs, and it will help me to document Py’n’APL capabilities. The Python side will be tested with pytest and the APL side will be tested with CITA, which is a Continuous Integration Tool for APL.

The code base also needs to be updated. Py’n’APL currently supports Python 2 up to Python 3.5. At the time of writing this blog post, Python 2 has been in end-of-life for more than 2 years and Python 3.7 is reaching end of life in a couple of months. In other words, there is no overlap between the original Python versions supported and the Python versions that an application should currently support. In addition, Dyalog has progressed from v16.0 to v18.2, and the new tools available with the later versions are also likely to be useful.

Another big thing that should be done (and that would pay high dividends) is to update the project management of the Python part of Py’n’APL. By using the appropriate tooling, we make it easier to clone the (open source) repository so that others can poke around, play with it, modify it, and/or contribute.

The First Commits

Let GitHub commit 4283176f4ffd7f1067f216c1459306cdbc49189a be the starting point of my documented journey. At this point in time, I have two handfuls of commits on the branch master that fixed a (simple) issue with a Python import and added the usage examples I showed at Dyalog ’21. So, what will my first commits look like?

Setting up Poetry

The first thing I decided to do was to set up Poetry to manage the packaging and dependencies of the Python-side of code. By using Poetry, isolating whatever I do to/with the Python code from all the other (Python) things I have on my computer becomes trivial and it makes it very easy to install the package pynapl on my machine.

Auto-Formatting the Source Code

Another thing that I did was to use black (which I added as a development dependency to Poetry) to auto-format all the Python code in the repository. I imagine that this might sound surprising if you come from a different world! But if you look at the commit in question, you will see that although that commit was a big one, the changes were only at the level of the structure of the source code; by using a tool like black, I can play with a code base that is consistently formatted and – most importantly – that is formatted like every other Python project I have taken a look at. This consistency in the Python world makes it easier to read code, because the structure of the code on the page is always the same. This means that there is one less thing for my brain to worry about, which my brain appreciates!

In a typical Python project using black, or any other formatter, the idea is that the formatter is used frequently so that the code always has that consistent formatting style; the idea is not to occasionally insert an artificial commit that is just auto-formatting.

Fixing (Star) Imports

The other major minor change that I made was fixing (star) imports across the Python source code. Star imports look like from module_name import * and are like )LOADing a whole workspace in APL – you will gain access to whatever is inside the workspace you loaded. In Python, star imports are typically discouraged because after a star import you have no idea what names you have available, nor do you know what comes from where, which can be confusing if you star imported multiple modules. Instead, if you need the tools foo and bar from the module module_name, you should import the module and use the tools as module_name.foo and module_name.bar, or import the specific names that you need: from module_name import foo, bar.

I therefore went through the Py’n’APL Python source code and eliminated all the star imports, replacing them by the specific imports that were needed. (OK, not quite all star imports; the tests still need to be reworked.) As well as fixing star imports, I also reordered the imports for consistency and removed imports that were no longer needed.

Python 2-Related Low-Hanging Fruit

To get started with my task of removing old Python 2 code, I decided to start with some basic trimming. For example, there were plenty of instances where the code included conditional assignments that depended on the major version of Python (2 or 3) that were supposed to homogenise the code, making it look as much as possible like Python 3. I could remove those because I know we will be running Python 3. Another fairly basic and inconsequential change I could make was removing the explicit inheriting from object when creating classes (this was needed in Python 2, but not in Python 3).

Explicit Type Checking and Duck Typing

Python is a dynamically-typed language, and sometimes you might need to make use of duck typing to ensure that you are working with the right kind of objects. At Dyalog Ltd we are very fond of ducks, but duck typing is something else entirely:

If it walks like a duck and if it quacks like a duck then it must be a duck.

In other words, in Python we tend to care more about what an object can do (its methods) than what the object is (its type). The Py’n’APL source code included many occurrences of the built-in type and I went through them, replacing them with isinstance to implement better duck typing.

What Happens Next?

These are some of the main changes that I have made so far; they happen to be mostly inconsequential and all on the Python side of the code. Of course, I won’t be able to maintain Py’n’APL by only making inconsequential changes, so more substantial changes will come next. I also need to take a look at the APL code and see what can and what needs to be done there. Although I haven’t looked at the APL code as much as at the Python code, I have a feeling that I will not need to make as many changes there. Fingers crossed!

This blog post covers (approximately) the changes included in this GitHub diff.

Enhanced Debugging with Function Keys – Evaluate selection

See also Enhanced Debugging with Function Keys.

When tracing through a complex dfn and reaching a guard (condition:result), I am often wary of tracing into that line because if the condition evaluates to 1 then the current function I’m tracing through will terminate and return the result, leading to me losing situational awareness. Normally, I’d select the condition expression, copy it, move to the session and execute the expression, so I can predict what will happen next. Can we automate this? Yes we can.

Now, I usually prefer the Windows IDE for my daily development, but this is actually a case where RIDE has neat feature that’s missing from the IDE (but if you keep reading, I’ll show you how to achieve a similar effect in the IDE). In RIDE, go to Edit ⇒ Preferences ⇒ Shortcuts (or simply click ⌨︎ at the right end of the language bar), then type the name of a function key you want to use for this purpose, followed by a space, for example “F6 ” for . You’ll see exactly one entry in the listing. In the input field, write “<VAL>” (without quotes):

I defined a simple function to test it with, and traced into that:

      ⎕VR⎕FX'f←{' '⍺∧⍵:''both''' '⍺∨⍵:''either''' '''neither''' '}'
     ∇ f←{
[1]        ⍺∧⍵:'both'
[2]        ⍺∨⍵:'either'
[3]        'neither'
[4]    }
     ∇ 
      f

Tracing into f
Upon reaching a guard, I select the condition:
Selecting the condition
And Press :
Pressing F6
Voilà!

Cool, but how about the IDE?

Right, the Windows IDE doesn’t support the VAL command code, but we can easily emulate it by combining multiple command codes and assigning them to an F-key using ⎕PFKEY.

What we need to do is:

  1. Copy the current selection
  2. Jump to the session
  3. Paste
  4. Execute
  5. Jump back again

Options ⇒ Configure… ⇒ Keyboard Shortcuts ⇒ Description gives that the command codes for “Copy”, “JumP between current window and session window”, and “Paste” are CP, JP, and PT. We use ER (you can find all but JP using the ]KeyPress user command too) to press . Here we go:

      'CP' 'JP' 'PT' 'ER' 'JP' ⎕PFKEY 6
┌──┬──┬──┬──┬──┐
│CP│JP│PT│ER│JP│
└──┴──┴──┴──┴──┘

Keep it so!

RIDE keeps its setting, but of course, I wouldn’t want to be bothered with setting this up for every IDE session. So here’s a trick to set up F-keys (or anything else for that matter). When Dyalog APL starts up, it will look for MyUCMDs\setup.dyalog in your Documents folder ($HOME/MyUCMDs/setup.dyalog on non-Windows). If this file contains a function named Setup, it will be run whenever APL starts:

      ∇Setup
[1]  '<F6> is: ','CP' 'JP' 'PT' 'ER' 'JP' ⎕PFKEY 6
[2]  ∇
      (⊂⎕NR'Setup')⎕NPUT'C:\Users\Adam.DYALOG\Documents\MyUCMDs\setup.dyalog'

And now, when I start APL:
Upon start

Dyalog ’18 Videos, Week 6

Happy New Year – and Welcome to the 6th week of Dyalog ’18 video releases!

If you enjoy geometry, 2019 starts with a couple of real treats; one which builds up to the use of complex numbers just before the end, and another which starts with them and moves on to Quaternions. Alternatively, if you think vectors and matrices containing imaginary numbers are a bit esoteric, what could be more “down to earth” than taking a look at various ways to efficiently extract data from Excel spreadsheets? Finally, we have a talk on a Theory of Everything, which will obviously interest everyone!

Returning to the maths: Nic Delcros asks a seemingly trivial question about the number of dimensions of a vector. As any APLer knows, a vector is a list of numbers and, therefore, has 1 dimension, but of course the numbers in a vector nearly always represent a structure of higher dimensionality. Nic takes us on an entertaining exploration of the case where the numbers represent a dynamic event, where one of the dimensions is time – punctuated with beautiful images.

Dieter Kilsch from the University of Applied Sciences (Technische Hochschule) in Bingen obviously enjoys teaching mathematics! In this talk, he actually managed to make me think that I had some insight into why the Irish mathematician William Hamilton invented the Hamiltonian number system (which is populated by Quaternions), and how it allows us to do algebra on points in a 3-dimensional space, similar to the way complex numbers work for 2 dimensions. For example, Quaternions can be used as a tool of thought and computation for image recognition!

Returning to the very real world, Richard Procter is back with an updated talk on “Excel Mining”, following on from his talk at Dyalog ’15 in Sicily. Like many of us, he frequently needs to load data which originates in Microsoft Excel into APL for processing – and sometimes write back to Excel. Richard has tried a variety of different techniques and provides a list of questions that might decide which technique to use in a given scenario (and performance measurements as well).

It should be no big surprise that John Daintree’s big TOE is not something he needs to take a shoe off to demonstrate. Rather, the Theory Of Everything is a unifying idea that might one day replace a large number of system functions, “root methods” and I-Beams which currently allow programmers to ask questions about the Universe that they are running in. The result will hopefully be a system that is more powerful, but simpler and much more self-documenting than the collection of tools that it would replace.

Summary of this week’s videos:

 

Dyalog ’18 Videos, Week 4

This week is mostly a deep dive into the new world of storing source code in text files rather than workspaces and other “binary” formats. However, in case that is not your cup of tea yet, we can offer you another talk by Marshall Lochbaum, who presents more amazing algorithms to make the very widely used primitive search functions ∊, ⍳ and ⍸ run faster than ever before. By combining non-branching algorithms with vector instructions and a technique known as Robin Hood Hashing, Marshall is able to drive a modern CPU close to the theoretical maximum throughput, and in many cases spend less than one nanosecond searching for each item of an array.

Source code in text files is the dominant theme, and we are fortunate enough to have three pioneers to show us the way: Paul Mansour, Gilgamesh Athoraya and Kai Jaeger.

Paul has been working on – and using – source code management systems for decades. Recently, his team have implemented a lightweight version of the Acre project management system, named Acre Desktop, based entirely on textual source files. Apart from having to start your day by ]Open-ing a project, rather than by )LOAD-ing a workspace, there are very few changes to how you would actually use APL – but now you have access to a huge collection of professional tools developed for programmers using other programming languages, such as GitHub.

One of the very significant advantages of the APL community starting to use common structures for source code – and projects – is that it becomes realistic to share tools and utilities.
Following on from Paul’s talk, Gilgamesh Athoraya demonstrates a prototype of an APL Package Manager (APM). The APM connects to a repository of packages written in APL and allows you to declare package depenedencies from a public or private repository. It also keeps tabs on the availability of new versions of dependencies, and allows you to easily update them when the time is right.

A package manager can only be successful if there are packages to be managed. Kai Jaeger has been an APL Toolsmith for a very long time, and made much of his work available via the APLTree. Now, Kai has transferred the contents of the APLTree to GitHub, making everything available as textual source. With a bit of luck, once the APM finds its legs, we’ll all be able to use Acre Desktop to define projects, Git[Hub/Lab] to manage the source, and APM to search for Kai’s tools and manage our dependencies on them!

Summary of this week’s videos: