Dyalog ’18 Videos, Week 2

Each week until early January, we will be releasing a selection of recordings of presentations from Dyalog’18, which was held in Belfast at the end of October 2018. Last week we kicked off with the opening keynote talks and the prize ceremony and acceptance speech by the winner of our annual problem-solving competition.

Just under half of the presentations at Dyalog User Meetings are by users who have volunteered – or sometimes been commandeered – to share stories about how they have used APL for fun or profit. These user stories provide significant motivation to the Dyalog team for future direction.

Aaron Hsu’s talk on “High Performance Tree Wrangling, the APL Way” is a pearl. Back in 2015 I gave a talk at Google on APL. One of the Google engineers asked about working with trees in APL and I was unable to give him a useful answer. Aaron is working on a compiler for APL, and trees that represent the code that is being compiled are his most important type of data structure.

In this talk Aaron demonstrates that APL is an elegant – and highly efficient – notation for working with trees, if you just pick the right representation!


Most of the talks at Dyalog User Meetings are fairly technical. The subject at the core of Ilaria Piccirilli’s talk – the fair pricing of financial instruments and subsequent evaluation of portfolios – is no exception. Mercifully, Ilaria spares us the details of the calculations – as she dryly notes, there is no “Fair Pricing for Dummies”. Instead, she offers humorous insights into the way her team used APL to deal with the explosion of computations required by regular additions to legislation requiring health checks – and the day that negative interest pulled the rug out from under most standard pricing calculations.

The other, slightly larger half of the talks at Dyalog Users meetings are by members of the Dyalog Team, talking about work that has recently been done on our products or presenting designs for future enhancements.


Adám Brudzewsky’s talk, titled Array Notation Mk III, is about a potential future extension to the APL language, which will make it possible to easily and clearly describe arrays of high rank, or with deeply nested structure, without using APL primitives to “construct” them, as is common practice today. In addition to making application code easier to read and write, a literal notation for data structures will make it easy to use text files to describe data structures which are essentially part of the source code of an application, and should be managed by a source code management system. As the name suggests, this work has been ongoing for some time, with the initial inspiration coming from a user presentation by Phil Last, back at Dyalog ’15 in Sicily. Watch the presentation and give us feedback on whether you think this idea is now sufficiently baked to become part of Dyalog APL, or we’ll need a “Mk IV” talk next year!


With the growth in usage of Dyalog APL under macOS and Linux – especially in server or cloud environments – the Dyalog Remote Integrated Development Environment is becoming a “mainstream” tool, rather than the curiosity that it was during the first few years of development. Our partners at Optima Systems are developing RIDE on Dyalog’s behalf, and Gilgamesh Athoraya is now the lead developer. In his talk on “RIDE 4.1 and Next Generation Integrations”, Gil talks first about significant new features and performance improvements to RIDE in 4.1 – and then continues to talk about how components of the RIDE technology may be re-purposed to provide APL add-ins for popular development frameworks like the new Microsoft VS Code.

Summary of this week’s videos:

Welcome to the Dyalog ’18 Videos!

Three weeks have gone by since we waved goodbye to the last Dyalog ’18 delegates in Belfast. We’ve had time to catch up on sleep, half of us have had colds and recovered from them. Jason Rivers and Richard Park have started mixing and improving the audio and video recordings, and we are ready to release the first group of processed videos.


Our plan is to release batches of 3-5 videos, with enough variety for everyone to find at least one topic of interest each week. We have not reviewed all of the material yet; there are always one or two where something went wrong and we are unable to publish the recordings (or the presenter asks that we refrain from making the talk public), but we do expect to be able to make the vast majority of the talks available over the next couple of months.

Each week, I’ll be doing my best to introduce each set with a blog entry: The first batch contains cleaned-up versions of the presentations that were streamed live from Belfast. The audio and video quality is significantly enhanced compared to the live stream, and the most confusing gaffes in my own live demo have been removed 😊.


As usual, the user meeting opened with the traditional trio of keynotes by Dyalog’s CEO Gitte Christensen, the CXO Morten Kromberg (that’s me) and CTO Jay Foad. Gitte introduces a couple of new faces at Dyalog, and the contest winners, so everyone can plan to buy the winners drinks during the week. Gitte then discusses high level direction – announcing our intention to make the Linux version available to download, and included in public Docker containers and Cloud VM images, with no questions asked.


My own session mostly consists of a live demo of the potential consequences of making Linux licences really easy to get hold of. In an imaginary conversation with a data scientist, I demonstrate the use of Dyalog APL to implement an (admittedly silly) analytical function, and subsequently make it available as a web service and via a web site, finally deplying it to the cloud using a set of Public Docker containers, without once installing Dyalog APL itself.


Jay Foad rounded Monday’s live stream off with a review of the features of the recently released version 17.0, before moving on to talk about the work that the development team is planning for versions 17.1 and 18.0, scheduled for the spring of 2019 and 2020, respectively.


In accordance with tradition, we also streamed the Prize Ceremony for the International Problem-Solving Competition and – often the most interesting talk of the year – the acceptance speech where this year’s winner talked about his code, and the experience of learning APL. This year’s winner did not let us down; it is amazing how quickly you can learn to write really, really good APL code!

Summary of this week’s videos:

Response to Feedback on Cut, Under and Merge

At Dyalog ’15 John Scholes and I presented proposals for future operators cut, under, and merge. Following the release of the video of this presentation, we received some feedback from a user. Our response to the feedback may be of wider interest.

  1. It’s early days yet for cut, under, and merge (they are tentatively planned for release in version 16.0), and one of the reasons for that is so that we can receive feedback from the user community. So thank you for your comments.
  2. We take our inspiration from many sources, and while it’s true that J has cut, under, and merge, cut saw the light of day in Rationalized APL (1983), under in Operators and Functions (1978), and merge in Rationalized APL (1983) again. And even before that, the ideas have been in mathematics or functional programming for many years.Wherever the inspiration came from, we don’t take them just on faith, just because they are in J.Even now, even in these early days, cut, under, and merge have undergone a vigorous internal debate. The discussion is informed by the experience in other contexts, J or otherwise.
  3. We are interested in what you think are the “few fundamental ideas expressed by a handful of mathematical symbols”, what you say is the definition of APL. From our perspective we have not violated and will not violate what we think are the fundamental ideas. But perhaps our set of fundamental ideas differ from yours?
  4. No one here is thinking of adopting the J spelling scheme (with the dots and colons).
  5. The Dyalog development team are in unanimous agreement that dfns, ⍺⍵-functions, are A Good Thing. As we see it, the recent and proposed additions to Dyalog APL have only enhanced the elegance and utility of dfns. For example, {(+⌿⍵)÷≢⍵}⍤r to compute the mean of not just vectors but subarrays of a specified rank, {⍺⍵}⌸ to compute the unique keys and the corresponding indices, {⍺(+⌿⍵)}⌸ to compute the unique keys and corresponding sums, and {⊂Cut ¯2⊢⍵,','}Cut ¯2 ⊢x~⎕av[4] to partition CSV x into its columns and cells.
  6. Cut, under, and merge are described and documented in discussion papers accompanying the video. If you have not already read them, they can be found in the Dyalog ’15 webpage and specifically here. Please tell us what more can be done as far as documentation, and what parts specifically are unclear.
  7. We have previously (and recently) examined the question of adding quaternions to Dyalog. Currently there are no plans to add them.
  8. Finally, examples of under from everyday life. Of course the processes involved can be composed, so that for example “dinner” can be:
    open the fridge
        take the food
    close the fridge

    but it can also be a more elaborate:

    take plate from cupboard
        open the fridge
            take the food
        close the fridge
     
        put food on plate
            open mouth
                put food in mouth
            close mouth
        clean plate
    put plate back into cupboard

    As remarked in the Dyalog ’15 presentation, once you are attuned to it you can see under everywhere, sometimes in subtle ways: Ashes to ashes, dust to dust.