Dyalog ’22 Day 5: Looking Forward

Links to recordings from this day of the user meeting are at the bottom of this post.

To begin the final day of the user meeting, Aaron Hsu presented some strategies and ways of thinking about processing data that come from text. While his specific focus is parsing APL code for the co-dfns compiler, he believes that a parallel data-flow approach will be useful across domains, for example, when processing JSON data, which is common in modern web communication.

Aaron talks about parsing data from text

Aaron talks about parsing data from text

Adám Brudzewsky, despite not being in the room, captivated us with a supremely convincing argument for why three primitives could be considered to be filling in “core language” gaps: the “Select” function (⍺⊇⍵) for elegant indexing; the “Depth” operator (F⍥k) as analogous to rank (F⍤k) for nested arrays; and the “Behind” (F⍛G) operator complementing other function compositions “Beside” (F∘G), “Atop” (F⍤G) and “Over” (F⍥G) to make tacit programming an even better extension of APL expressivity.

Adám proposes a new "behind" function composition operator

Adám proposes a new “behind” function composition operator

It was exciting to see a return for APL to its birthplace in academia. Jesús Galán López of the Technical Univerity of Delft told us his plan to use APL in metallurgical models and analysis of experimental data. He hopes to use Dyalog APL as a central connecting piece in education, science, engineering and research. This is one of the first projects to eventually be supported by the APL fund mentioned by Stine on Monday.

Jesús with a typical data analysis example in materials science

Jesús with a typical data analysis example in materials science

A living legend in APL education, Raymond P. Polivka presented his latest book aimed at high school students with no programming knowledge whatsoever. The clearly structured modules and exercises will be an invaluable resource for teachers and recruiters alike, and they have been developed with both a lifetime of experience teaching APL as well as very recent classes with real high school students. He is asking for feedback on a draft available at aplclass.com/book.

Ray Polivka introduces his new book aimed at high school students

Ray Polivka introduces his new book aimed at high school students

For the last talk of the user meeting, Rich Park took us on a whirlwind tour of the media and training materials being developed at the moment. He also commented on some of the work that he believes is needed to tie these resources together and make them more discoverable, and what might be done to improve the experience for new users trying to find things out for themselves.

Rich Park talks about media, training and outreach at Dyalog

Rich Park talks about media, training and outreach at Dyalog

In the afternoon we ended the user meeting with some workshops. We got to know about performance considerations in the context of the Dyalog interpreter with Aaron Hsu and Karter Kooner. There was a chance to get hands-on experience using one of the asynchronous, concurrent programming methods in Dyalog: Futures and Isolates with Morten Kromberg and John Daintree. Josh David and Brian Becker taught us about sharing APL tools by creating and publishing packages using Tatin.

It has truly been an incredible experience returning to in-person user meetings. The sense of community felt in Olhão this year was far beyond what most people predicted, not just in light of separation over the last two years, but also in looking forward to an exciting future for APL and Dyalog.

Today’s presentations (links to recordings will be added as they become available):

Dyalog ’22 Day 4: Celebrations

Links to recordings from this day of the user meeting are at the bottom of this post.

The third day of presentations kicked off with Brian Becker running us through the gauntlet of setting up and deploying web services. Initial setup with Jarvis and Docker containers seems like an absolute breeze. However, the later stages configuring cloud services can be more fiddly.

Brian Becker talks about creating and deploying web services.

Brian Becker talks about creating and deploying web services.

Stephen Mansour of Misericordia University then gave us the hot tips for producing hot tubs. His new system TAMPA (Taming Mathematical Programming in APL) can be used to optimise some decision (e.g. how many hot tubs to produce) for some objective function (e.g. maximise profit) according to some constraints (e.g. resources available). The use of APL in TAMPA makes a near 1-to-1 translation of linear programming expressions into executable code.

Stephen Mansour explains the TAMPA mathematical programming framework.

Stephen Mansour explains the TAMPA mathematical programming framework.

We then got to hear some strong but convincing opinions about graphical user interfaces from Norbert Jurkiewicz, who told us about how The Carlisle Group has been incrementally integrating the HTMLRenderer and “the triad” of HTML, CSS and JavaScript into their systems. He championed the benefit of hiring external developers, in addition to the portability of using web-technologies, for graphical front ends.

Norbert Jurkiewicz gives his views on using web stack for front end development.

Norbert Jurkiewicz gives his views on using the web stack for front end development.

Neither Gitte Christensen nor Brian Becker are shy about saying that some of their favourite parts of every user meeting are the competition presentations. Luckily for us, both this year’s student winner and professional winner came to share their experiences about participating.

Professional winner Michael Higginson had actually been a kdb+ and q programmer for many years before recently deciding to expand his horizons with APL. He gave a fantastic breakdown of his thought process in solving both an easier problem which he found built his confidence, and then the notorious problem 6 on interpreting human-readable dates and times.

Michael Higginson takes us through his array programming journey.

Michael Higginson takes us through his array programming journey.

The audience could definitely empathise with all of the reasons given by student winner Tzu-Ching Lee as to why he likes APL: the glyphs; the concise syntax; operators; and algorithms as primitives. Alongside his excellent walkthroughs of two of his solutions, he had brilliant ideas for extending the problem description for the Base85 encoding/decoding problem once he noticed additional symmetry which could be expressed in his solutions.

Student winner Tzu-Ching Lee presents his winning solutions to the APL Problem Solving Competition.

Student winner Tzu-Ching Lee presents his winning solutions to the APL Problem Solving Competition.

In the afternoon, we took a coach about an hour away to the Quinta dos Vales vineyards and winery. We were treated to a tour of the winery, learning about the fermentation process; what goes into deciding whether to make single grape or blended wines; and the use of wooden barrels to imbibe additional flavour. Afterwards, we were split into teams and challenged to make our best and favourite blends of wines from three grapes. According to the judges, a majority Cabernet Sauvignon, with about a third Aragonês and just ten percent Touriga Nacional makes for the most delicious blend of tannins and spices. Later that evening, we enjoyed a delicious Portuguese churrasco – or barbeque.

Delegates enjoy the afternoon sun at Quinta dos Vales winery in the Algarve, Portugal.

Delegates enjoy the afternoon sun at Quinta dos Vales winery in the Algarve, Portugal.

Congratulations to the winners of the APL Problem Solving Competition, and congratulations also to the winners of the wine blending competition!

Today’s presentations (links to recordings will be added as they become available):

Dyalog ’22 Day 3: Automation, architecture and performance

Links to recordings from this day of the user meeting are at the bottom of this post.

Automation, architecture and performance were throughlines of the second day of presentations. Lars Stampe Villadsen from SimCorp A/S provided some advice on how to write tests and gave a live demonstration of a small testing framework together with continuous integration tools which run the test suite every time a change is committed to the project – useful for when we forget to run some tests locally.

Lars Stampe Villadsen talks about testing.

Lars Stampe Villadsen talks about testing.

Norbert Jurkiewicz presented his “10,000 ft overview” of automation processes used by The Carlisle Group and described some of the complexities of building distributables according to varying requirements for different customers. He discussed some of the cost considerations to be made when using Amazon Web Services (or any cloud computing service) to leverage lots of computing power to do builds quickly.

Norbert Jurkiewicz discusses automation architecture.

Norbert Jurkiewicz discusses automation architecture.

This was followed by Michael Baas with a similar issue viewed from a different angle. He talked about using the ]DTest testing framework to produce code coverage reports – showing which lines of code had actually been executed when running a test suite – and also automating testing across different platforms and versions of the Dyalog interpreter.

Michael Baas shows us the ]DTest framework

Michael Baas shows us the ]DTest framework

Changing tack, we heard the story of getting to grips with semi-global variables in a multithreaded application. Elena Pavarotti of SimCorp Italiana talked about her experience trying to imagine clearly what a complex system is doing so that it is easier to reliably refactor and adapt it to talk to new external systems.

Elena Pavarotti on managing complexity in SimCorp Sofia.

Elena Pavarotti on managing complexity in SimCorp Sofia.

Rodrigo Girão Serrão then told us how himself and Aaron Hsu had implemented a U-Net Convolutional Neural Network from scratch in APL. The comparisons between their implementation and industry standard libraries was very interesting, and then seeing the mapping from diagrams to code showed us how the APL becomes a natural way to express the data flow involved in the system.

Rodrigo Girão Serrão walks us through the U-net CNN in APL.

Rodrigo Girão Serrão walks us through the U-net CNN in APL.

Delving deeper into more intellectual musings, Justin Dowdy of Semantic Arts – also known for his work on the April APL to Lisp compiler and the May bridge between Dyalog and Clojure – drew some interesting analogies between the Resource Description Framework, used for representing data relationships as ontologies, and points raised in Iverson’s paper Notation as a Tool of Thought.

Justin Dowdy compares Notation as a Tool of Thought and concepts of Data Semantics.

Justin Dowdy compares Notation as a Tool of Thought and concepts of Data Semantics.

Juuso Haavisto is a DPhil student at the University of Oxford. He brought up some hot topics in computer science academia – static analysis, rank polymorphism and scheduling for multi-core systems – that he believes can be tackled effectively if we can learn to make the computer think a bit more like an APLer.

Juuso Haavisto with three hot topics in computer science academia.

Juuso Haavisto with three hot topics in computer science academia.

The theme of performance continued when Vali-Matti Jantunen from Statistics Finland compared the performance of some short APL phrases across several versions of Dyalog. Of course, this can be important in an application like PxEdit which processes thousands of text files; a fraction of a second difference processing a single file can lead to several minutes across a job.

Vali-Matti Jantunen on Dyalog performance across versions.

Vali-Matti Jantunen on Dyalog performance across versions.

We hope Veli-Matti will forgive us for the performance regressions found in v18.2 (although not slower than v17.1). However, for us it was no surprise, and next Karta Kooner reflected on some of the assumptions made when changes, intended to improve performance, had been implemented over the lifetime of Dyalog APL. He is looking for volunteers to run a version of the interpreter which can gather usage statistics, so please get in touch if you can do this.

Karta Kooner analyses performance in the interpreter.

Karta Kooner analyses performance in the interpreter.

Presenting the final features of his special Dyalog ’22 Conference Edition of the interpreter, John Daintree talked about the state of asynchronous programming in the interpreter now. Eventually he showed us an interface, in the from of an ⎕AWAIT system function, to unify Spawn (F&⍵) threads, .NET Tasks and Futures and Isolates.

John Daintree with some ideas about asynchronous programming.

John Daintree with some ideas about asynchronous programming.

Finally, Aaron Hsu’s Co-dfns report showed a new public API for the parser which could be useful for static analysis of existing code and better error reporting within co-dfns. The new code generator written in APL opens the door to targeting more platforms more easily in the future.

Aaron Hsu presents an update on co-dfns.

Aaron Hsu presents an update on co-dfns.

Today’s presentations (links to recordings will be added as they become available):

Dyalog ’22 Day 2: Welcome Back to User Meetings

Links to recordings from this day of the user meeting are at the bottom of this post.

We were welcomed to the 2nd day of the user meeting by managing director (CEO) Gitte Christensen. Everybody could appreciate the emotion and sentiment when she noted the distinct “APL Hum” apparent in the atmosphere at this year’s event.

Managing Director Gitte Christensen welcomes us to the user meeting.

Managing Director Gitte Christensen welcomes us to the user meeting.

In some exciting news, Stine Kromberg announced that Dyalog’s board of directors have approved an APL fund for education and science. She also made a call to action for members of the APL community to present their stories to the wider world.

Stine Kromberg announces the APL fund.

Stine Kromberg announces the APL fund

Morten Kromberg’s road map provided an overview of the multitude of projects and requirements Dyalog is to address in the upcoming years. From the need for new APLers to be recruited to replace those career programmers leaving the fold, to developing tools and training resources that can help all users navigate the modern, complex world of interconnected software development and making the user experience more pleasant and consistent across different platforms.

Morten Kromberg presents the roadmap of activity at Dyalog

Morten Kromberg presents the roadmap of activity at Dyalog.

Morten later returned to speak about the status of project and package managers for Dyalog. While Carlisle Group’s Dado is an excellent recommendation for those who can adopt its tools and workflow, Dyalog Ltd has employed the help of APLTeam to develop the more agnostic Tatin package manager and Cider project manager, with the hopes of providing a more flexible and extensible set of tools to meet a wider range of needs.

In a double bill of talks from John Daintree, we were shown a special 2022 Conference Edition of the Dyalog interpreter which has features stemming from both internal discussions and projects and our users’ requests and ideas. He reiterated some of the complications of input and output in Dyalog, but showed how the conference edition can more intelligently handle output from the various sources – for example error messages, quad output ⎕←message and output from system commands.

In his second talk, he demonstrated a system for piece-by-piece debugging of inline APL code and functions. We think you will agree that being able to step through the evaluation of a single line of APL has massive implications for teaching, writing and understanding APL.

John Daintree demonstrates inline "token by token" debugging

John Daintree demonstrates inline “token by token” debugging.

Charles Brenner from DNA-View took us on a journey of his discovery of a method of performing numerical integrations in high dimensional spaces – using a kind of Riemann summation (analogous to the rectangle or trapezium methods) using simplexes (multidimensional generalisation of triangle and tetrahedrons).

Charles Brenner talks about numerical integration.

Charles Brenner talks about numerical integration

Gilgamesh Athoraya presented a recent project by Tiamatica AB to migrate a process planning system from mainframe APL2 to Dyalog in the cloud. This large undertaking began with creating a bridge from Dyalog to APL2, and the challenge continues as code must be migrate to Dyalog while being able to fold in updates made by the APL2 team still maintaining the current software.

Gilgamesh Athoraya talks about migrating APL2 mainframe to Dyalog in the cloud

Gilgamesh Athoraya talks about migrating APL2 mainframe to Dyalog in the cloud.

Another APL2 conversion, Mark Wolfson of BIG provided some fascinating insights into the world of Jewelers in North America. Many of these business have managed to resist the trend of becoming parts of large conglomerates and have stayed as small, often family owned businesses. This has meant that they usually don’t have sophisticated software for managing their sales and inventories. Mark’s system uses APL to ingest data from an incredible variety of formats and provide insights to help these businesses thrive.

Mark Wolfson gives insight into the world of North American Jewellery retailers.

Mark Wolfson gives insight into the world of North American Jewellery retailers.

Unfortunately, Kimmo Linna from Finnair could not join us in person due to his obligations as a pilot. Fortunately for us, his time away from the cockpit during the pandemic led to him developing useful tools and workflows that he was able to show us remotely via video feed. He is using DuckDB to store data and Vega-lite to present it visually, all being driven from Dyalog Jupyter notebooks and connected using bridges that he has published and made available for free on GitHub.

Kimmo Linna presents his Dyalog/DuckDB/Vega-lite/Jupyter workflow

Kimmo Linna presents his Dyalog/DuckDB/Vega-lite/Jupyter workflow

To finish the day, new recruit on the development team Peter Mikkelsen talked about a personal project from before he joined Dyalog: an APL implementation for the Plan9 research operating system. As he explained, it may be lacking performance, documentation and even users, but we could see from his approach and language design choices that he is someone who can bring great ideas to the interpreter development team.

Peter Mikkelsen shows message passing in APL9

Peter Mikkelsen shows message passing in APL9

Today’s presentations (links to recordings will be added as they become available):

Dyalog ’22 Day 1: Welcome to Sunny Olhão!

Arriving to meet the warm sea air is already a refreshing change as this year’s long, hot summer started to close in the weeks leading up to this year’s user meeting. Of course, for some with less far to travel, the warmth is a familiar comfort.

Apartments at the Real Marina in Olhão, Portugal

Apartments at the Real Marina in Olhão, Portugal

As delegates arrived on Sunday night we were treated to a lovely surprise – a birthday cake to celebrate Tony Corso’s birthday! Happy Birthday, Tony!

Today we kicked off the user meeting with workshops, giving delegates hands-on experience with a range of Dyalog offerings for application development.

Attendees of Rich Park and Rodrigo Girão Serrão’s language workshops explored different ways to express ideas in APL using both tried and true idioms in the morning and newer language features in the afternoon – they compared expressiveness and performance implications in difference circumstances.

Morten and Josh got participants up to speed with using and maintaining code which lives outside of your workspace in text files, while Richard Smith and Bjørn Christensen showed how to manage and interact with data from outside of the workspace.

Brian Becker guided delegates into the world of SAAS (Software As A Service). It is encouraging to see how straightforward having APL code talk to the outside world can be. As usual, however, the complexities reveal themselves as you delve deeper in to specific use cases and circumstances.

Morten presents a Dyalog workshop

Morten and Josh show how to store source code as text files using Link

Without a doubt, getting to meet our users face to face has already proved to be the greatest enjoyment of the meeting so far. Some familiar faces long since last seen, and others only over a screen. Discussing ideas face to face still retains a value impossible to quantify.

Overall a fantastic beginning to the week and we cannot wait to experience the rest of this week!

APL Seeds ’22: Tuesday 29 March

On Tuesday 29 March we hosted APL Seeds ’22, the second annual online event for new and prospective users of APL (although everyone was welcome). Once again we were delighted to see that the majority of registrants had little to no APL experience; it feels like we get the chance to offer that same sense of discovery we felt when first learning about the language.

APL Seeds ’22 began with an introduction from Dyalog Ltd’s managing director, Gitte Christensen, in which she described her experiences of seeing APL enable people who had real problems to solve and showed some of what Dyalog provides in terms of the tools and interfaces that people might expect from a modern software development stack. Gitte explained the new Basic Licence, which is another step forward in Dyalog Ltd’s aim to bring APL to a wider audience. The Basic Licence allows non-commercial distribution of Dyalog along with APL-based solutions under the terms of the Royalty-Based Run-Time Licence, which will apply as the default run-time licence (see Prices and Licences for more information on Basic Licences). She also described some customer use cases, some of which might surprise newcomers to this language. Rich Park finished the introduction by pointing out where you can find more APL content, especially if you’re just getting started. For example, our tips for beginners includes things that might not be obvious when you first start the interpreter or read introductory books. The video description for the recording of this presentation contains many useful links!

APL is an executable notation and tool of thought which enables people with good ideas to bring them to life with computers

Gitte opens the event with her thoughts on “What is APL?”, including the short expression 1 2 3 + 4 5 6 that started the journey which eventually brought her to Dyalog.

Rich then presented a basic introduction to APL, showing the benefits of a symbolic notation for programming as well as demonstrating how to put together simple building blocks to build a function. After introducing the basic syntax, right-to-left precedence, and the generality of APL operators, along with a handful of symbols including the famous outer product (⍺∘.F⍵) and array indexing (⍺[⍵]), he walked through constructing a function that visualised the probability distribution for sums of rolling two N-sided dice.

A screenshot of an APL function to do some simple statistical computation and display the output using characters in the APL REPL

The Dist function uses just a handful of APL constructs to create a visualisation of a simple statistical distribution.

Stefan Kruger, author of the online book Learning APL, took us on an exploration of bioinformatics problems – a popular topic for the annual APL Problem Solving Competition. He described what a “k-mer” is (a chunk of DNA of a particular length), and compared different techniques for cutting up a text vector to isolate them from a DNA string, including a windowed-reduction (⍺F/⍵) and its generalised cousin, the stencil operator ((⍺⍺⌺⍵⍵)⍵), and our old friends the outer product and array indexing. Finally, Stefan looked at three approaches to doing some simple statistics in the Rosalind challenge “Computing GC Content“. To our delight, a new user who was in attendance commented that they learned new expressions and idioms that they had not seen before.

Stefan compares 3 functions to split a string into lengh-4 substrings

Stefan compares 3 functions to split a string into lengh-4 substrings

Andrew Sengul presented a more involved example, April. The April APL Compiler is a new entry in the APL field, compiling a subset of APL into the Common Lisp language and allowing APL functions to easily be used within Common Lisp programs. Andrew gave a concise history comparing Lisp and APL. He then gave a small introduction to using Lisp to write a “macro” (code that generates other code) before giving a glimpse into the implementation, architecture and design of APL in Common Lisp, as well as code that combines APL and Lisp to create visualisations. He showed a visualisation that used APL to both run Conway’s Game of Life and apply convolution kernels to show the state of cells over time. We loved seeing how he has been using April in the visual art installation Bloxl (a collection of computer-controlled light-up blocks used at events for a stunning visual effect). Andrew concluded his presentation with a demonstration of April code implementing a “falling block game”, and a video of that game in action.

Andrew gives a comparative overview of the histories of the APL and Lisp programming languages

Andrew gives a comparative overview of the histories of the APL and Lisp programming languages

Finally, there was a live recording of an episode of Array Cast, a semi-weekly podcast about array languages. From the regular panel of presenters were self-proclaimed J enthusiast Bob Therriault, Kx Librarian Stephen Taylor, Dyalog tools developer and life-long APL programmer Adám Brudzewsky, and professional C++ developer and programming language fanboy Conor Hoekstra. They were joined by a very special panel of guests: speakers from the event Gitte Christensen, Rich Park, Stefan Kruger, and Andrew Sengul, and well-known APLers Aaron Hsu and Rodrigo Girão Serrão.

The discussion began with the common beginners’ questions of keyboards, typing APL and whether you really need stickers, keycaps or a whole dedicated APL keyboard to use APL. On the topic of actually learning APL, there were mentions of even more books and other resources, including YouTube channels run by some of the podcast panellists and guests. All relevant links are included in the show notes for the episode, and you can listen to the episode on arraycast.com.

Taking a more technical turn, there was discussion of the balance between code clarity and performance. Should code be clear unless absolutely performance critical? Or is it possible to have both, where the clearer encodings and approaches are also the fastest? The episoded was capped off nicely, in response to a question from the audience, with Gitte offering her perspective on how APL can help a data analyst or engineer.

In the informal meet-up after the talks, Andrew configured a simple interface to the aforementioned “falling block puzzle game”, in which participants could control the game and see their moves played out on a Bloxl wall streamed in real (if a bit delayed) time. The players tried their best but were ultimately thwarted by the interface of clicking buttons using Zoom shared controls!

To those who attended, we hope you found the event enjoyable. Relevant materials have been uploaded to the APL Seeds ’22 webpage, including links to recordings of the presentations on dyalog.tv.