Tools Architect Brian Becker started today’s presentations with an overview of the tools provided by Dyalog Ltd. to help develop applications. He also explained the plans to improve documentation and tutorials and have more comprehensive testing, especially as our tools are supported across different platforms, architectures, and deployment scenarios. He then demonstrated use of HttpCommand to issue HTTP requests to web services, and announced the new WebSocket Server that he has been working on to facilitate asynchronous communication between Dyalog and non-APL applications.

Finn Flug discusses migrating an APL+Win application to Dyalog

Finn Flug presented a project recently undertaken to migrate a service created with APL+Win into a Jarvis-based web service with a browser-based user interface – he had some very useful advice on thread-safe error handling and implementing logging when deploying with containers because the containers disappear if the application crashes. To his satisfaction, he found that working with Dyalog tools allowed him to drop existing code into Dyalog, make a few minor tweaks, and discover that everything just works.

Ron Murray gave a demonstration of the results of his work migrating Dyalog to run on 64-bit ARM processors. He showed Dyalog running on a Raspberry Pi 4 as well as an Amazon ARM chip on Amazon Web Services (AWS). Of course, for the APL user on the new platform, it’s just APL and it works just the same.

Claus Madsen then discussed his financial application, which is deployed on AWS and needs constant up-time. He gave us a glimpse into his architecture: launching processes that log calls to the application so that when a customer reports an issue they can replicate the function calls exactly. Finally he demonstrated his continuous integration deployment workflow.

Elena Paviotti and Gianna Folladore of SimCorp Italiana entertained us with various experiences they have had when working with files and with people using files. This included problems such as when you ask for a CSV file but a naïve user has renamed a Microsoft Excel file – it looks like a CSV when opened in Excel, but is complete nonsense when imported as text! Regional settings for dates and numbers also provide a challenge, so dealing with these types of issues takes curiosity, collaboration, patience, and even more patience.

Gianna Folladore (left) and Elena Paviotti (right) share some funny stories of people and files

With an exciting new application, newcomer to APL Marcos Frenkel presented an introduction to the weird world of quantum computing. The current state of quantum programming languages are very implementation-specific, so he and his team from the University of Illinois are exploring APL as a high level way to express quantum algorithms. His representation of states as 1-column matrices and gates as square matrices allows application of gates to states to be simple matrix multiplication (+.×), which is natural and easy to explain to colleagues in the field.

Marcos Frenkel talks about quantum computing

The buzz of excitement at new scientific research with APL continued as Jesús Galan Lopez gave us an update on his investigations in metallurgy. He expressed how, in research, usually scientists are themselves the developers and they simply want to know “can APL solve my problem?” He has been pleasantly surprised by the reception to APL when he has talked about it to “whoever will listen”.

One of our two interns from this summer, Asher Harvey-Smith, described the work that he did during his internship; he started by implementing an extension to import data from text files as character matrices, then he addressed some issues in the Jupyter kernel that he found while writing some notebooks to teach linear algebra to other students at his university. Since returning to University he has seen great benefit to teaching an already familiar topic re-framed with the APL notation, and he anticipates a great opportunity using APL in that way for teaching in the future.

Asher Harvey-Smith relays his experiences teaching APL to fellow university students

Reflecting Mark Wolfon’s message yesterday, Stefan Kruger demonstrated an interface between Dyalog and Kafka, a publish-subscribe message streaming platform for managing inter-process communications in complex systems. In a modern, distributed, micro-service landscape, it is often the language of choice and, as Stefan sees it, if Dyalog can talk to Kafka then it can talk to anything that Kakfa talks to.

One issue raised in Stefan’s talk was that he had to leave APL and write a thin wrapper in C# to talk to the .NET Kafka client. John Daintree explained that this is because of a feature of C# called “generics” – generic methods and classes. If you live in C# world you probably already know about generics, and if not then you just might benefit from some awareness as many .NET packages in NuGet are only accessible using generics. Today that might require some C# code, but soon it will be all possible from APL.

When using the key operator (⌸) to do things with grouped data, it can be awkward to deal with absent values, sort the result, or query only a subset of the data. Adám Brudzewsky’s proposal for an array left operand that specifies the “dictionary” to key addresses all of these issues. The main issue currently is that this derived function returns a nested vector of grouped items, and the user must then apply their function on each grouping, but he had ideas on how to recognise and optimise this pattern.

Uncle Andy (Shiers) returned to the fireside (with great effect thanks to Jason!) to tell us about some of the backroom processes and lesser-known features and aspects of Dyalog. While a heartfelt goodbye to Geoff Streeter (who retired earlier this year) was garnished with a story about his erratic driving, we got some very useful advice about making lightweight Dyalog docker containers by removing help and sample workspaces. We also learned that 819⌶ is being removed in version 20.0 (use ⎕C instead!) and that version 19.0 will be the last version produced for intel-based macOS. He also introduced us to the new QA dev, Aarush, who we all had the pleasure to meet for the first time in person this week!

Today’s presentations:

• D09: Dyalog Tools Update – Brian Becker
• U05: Converting a COM Server to a Jarvis-based Web Service – Finn Flug
• D10: Running Dyalog on ARM64 Machines – Ron Murray
• U06: Dyalog, AWS, Jarvis, Docker…What’s Not to Like? – Claus Madsen
• U07: Once Upon A File – Elena Paviotti and Gianna Folladore
• U08: quAPL – A Quantum Computing Library in APL – Marcos Frenkel
• U09: APL and Metallurgy – Jesús Galán López
• U10: Teaching Linear Algebra with APL – Asher Harvey-Smith
• D11: Dyalog + Kafka = True? – Stefan Kruger
• D12: Dyalog Version 20.0 – Part 2 – John Daintree
• D13: Giving Key a Vocabulary – Adám Brudzewsky
• D14: The Return of Uncle Andy’s Fireside Chat – Andy Shiers

Follow

The weather improved today – the sunshine was a refreshing feeling as we awoke to a new day of forward-thinking presentations.

As usual, we began with Gitte Christensen’s welcoming address in which she spoke of the exciting new growth happening at Dyalog Ltd. Five new employees have joined since we met in Portugal last year for Dyalog ’22, addressing our increasing needs for tools, administration, IT support, testing, and marketing. In addition, approximately a quarter of the delegates at Dyalog ’23 are attending for the first time. However, we also said goodbye to Geoff Streeter, one of the original implementers of Dyalog, as he retired this year (read about Geoff and his charity work in this blog post “Not All Heroes Wear Capes”). Gitte herself is stepping down from managing the company at the end of the year, and has been spending the last couple of years training her replacement as CEO, Stine, to take over in January 2024.

Gitte Christensen welcomes us to the user meeting for her last time as CEO

There are two large projects that Gitte has been championing before her retirement. The first is the formation of the APL Trust Fund – a charitable foundation to enable individuals and companies who have enjoyed and benefited from APL to pay it forward with a fund dedicated to science and technology research, and in particular that which can help promote the use and development of APL. The second is a new website for Dyalog Ltd, which is progressing well; she showed us a preview of the home page and a searchable library of APL videos.

Morten Kromberg then presented his insight into what this rapidly growing team is up to. After reflecting on the milestones of historic Dyalog development, such as the introduction of namespaces and the Win32 GUI interface, he talked about many of the ongoing projects that will be presented this week as well as his hopes for version 20.0 and beyond. For example, Dyalog on 64-bit ARM processors, academic collaborations in quantum computing, metallurgy and teaching; tools for asynchronous programming; package management; performance improvements, an application health monitor, and much, much more.

Morten Kromberg gives an overview of new development work happening at Dyalog Ltd

Next, John Daintree presented an update to the so-called Token-by-Token (TBT) Tracing – or is it primitive debugging? or should it be called “Bit-by-Bit Poking”?! The ability to do fine-grained tracing function-by-function within expressions clearly has incredible potential for development, teaching, and learning APL. Those with a tacit aversion can see exactly how derived functions and trains are executed, and everyone can benefit from seeing the values of variables in the debugger as expressions are traced.

Then Adám Brudzewsky presented his ideas on how to address a range of issues when you want to assign a variable but the name itself is variable – this is often a problem when using namespaces as dictionaries, for example. Although possible, he showed how incredibly unwieldy it currently is to do this with good performance. He proposed system functions for getting name-value pairs (⎕NV), getting a name (⎕NG) and setting a name (⎕NS). If you recognise the last one, luckily his proposal is a consistent extension of the current ⎕NS system function.

Peter Mikkelsen had another system function proposal as he elucidated current issues with the way we call shell commands from Dyalog with ⎕SH and ⎕CMD. His proposed new system function might include features such as whether to error on a non-zero exit code and ways to redirect input and output. This proposal is quite concrete, and if all goes well it might see its way into version 20.0.

Acknowledging that it might be of little interest to some users and yet deeply fascinating for (and requested by) others, Richard Smith gave us a privileged view into the technical content of the workspace and how memory allocation works. Essential viewing for users concerned with performance, the “pocket allocation algorithm” shows what really happens when a WS FULL error occurs.

John Daintree and Richard Smith discuss the internal workings of the interpreter

Furthering the discussion of interpreter performance, Karta Kooner continued the analysis he presented last year of the performance of set functions that use search algorithms, such as membership ∊ and index-of ⍳. His heat maps of performance using different data types with arrays of sizes ranging up to 1,000 or 20,000 elements, showed v18.0 having unexpected degradation in many cases. Of course, it is significantly faster in some cases, so Karta intends to address the algorithm selection issues and then re-introduce successful algorithms from v18.0.

On the topic of concurrent programming, Stig Nielsen of SimCorp demonstrated using .NET tasks to implement “APL Worker Bees”. By exporting an APL class as a .NET assembly, a main APL process can be used to control many interpreters working in parallel across all of the cores of the machine. In contrast to isolates, these are several APL interpreters running in the same machine process, which Stig needed when migrating his specific application to the cloud.

Mark Wolfson

Mark Wolfson returned to the stage after his fascinating presentation last year in Portugal to talk about his efforts streamlining a complex data processing and analysis pipeline. Previously, he had used C# tools to ingest data from over 40 different software systems used by U.S. jewellery retailers and some APL “black boxes” to normalise the data. He has been able to replace all of that with Dyalog tools, allowing him to rapidly make changes that previously took days, weeks, or months. He has also shown these tools to members of his C# team and said that they really resonated and he now has C# programmers wanting to learn APL! Mark urges the APL community to promote APL as an excellent “glue” language to bind together the already-familiar tools used by programmers.

Kai Jaeger

Of course, the APL world still needs its own tools. Kai Jaeger has been developing the APLTree library for many years. We saw a brief history of Kai’s library of tools before he provided an update, including the move to make them available on the Tatin package manager that he has been developing for Dyalog. There was news of specific updates to packages such as FiRe (Find and Replace), which is a tool for searching and replacing within code and names in a Dyalog workspace, and the addition of new packages for reporting code coverage during testing, an FTP client, accessing the GitHub API, and using SMTP for email. He also introduced the Launchy application, which is useful for users when starting Dyalog when they have multiple versions installed.

In the last talk of the day, Morten shared the status of package management in Dyalog more broadly. In exciting news, he demonstarted how the Cider project manager now supports adding both Tatin and NuGet dependencies to an APL project, making it easy to use the dozens of APL packages or hundreds of thousands of .NET packages.

After dinner we were treated to some light entertainment in the form of a remote presentation by Ray Cannon, who has been experimenting with synthesising music from APL. Hearing the development of sounds from simple sine waves through to added overtones creating a fairly convincing organ was quite fascinating, and the synchronised animation of falling piano keys to J.S.Bach’s Toccata and Fugue in D minor was magical.

Today’s presentations:

• D01: Welcome to Dyalog ’23 – Gitte Christensen
• D02: The Road Ahead – Morten Kromberg
• D03: Dyalog Version 20.0 – Part 1 – John Daintree
• D04: Setting and Getting Variable Values – Adám Brudzewsky
• D05: Revisiting ⎕SH and ⎕CMD – Peter Mikkelsen
• D06: An Introduction to the Workspace – Richard Smith
• D07: Future Performance Improvements in Set Functions – Karta Kooner
• U01: APL Worker Bees – Another Way to Run APL Tasks in Parallel – Stig Nielsen
• U02: Transforming and Streamlining a Complex Development Process using Dyalog – Mark Wolfson
• U03: The Evolution of APLTree and APL-cation – Kai Jaeger
• D08: Using Packages – Morten Kromberg
• U04: Music: APL vs APS (A Pointy Stick) – Ray Cannon

Follow

We have returned to the (very windy this time!) shores of Elsinore in Denmark for our annual user meeting, Dyalog ’23!

Today saw the usual range workshops in which everybody could get involved learning the intricacies of a range of topics with the support of experts from Team Dyalog and friends.

In the morning, users in part 1 of the Projects and Packages workshop started consuming APL packages using the Tatin package manager. While many existing users have gone decades without these very convenient additions, we see packages as an important part of APL’s future.

In the Tacit Techniques workshop, participants learned the basics of function composition operators and function train syntax. They practised spotting patterns to make explicit code more succinct, and even those who only started using APL earlier this year had great success converting explicit functions into those that do not mention their arguments.

Meanwhile, users who wanted a challenge tackled an advanced workshop focussing on common pitfalls when writing code that performs well.

Projects and Packages workshop with Morten Kromberg, Kai Jaeger, and Gilgamesh Athoraya

In the afternoon, we hit the rate limit on the GitHub API during the session on web services, as presumably everyone on the venue’s Wi-Fi network had the same I.P. address! Fortunately, after each user successfully configured a web service on their own machine, everyone could experiment as much as they wanted to.

In part 2 of the Projects and Packages workshop, participants managed to publish their own APL packages to the Tatin test registry for others to use. Now that they’ve had a taste, we hope to see more become available on the main Tatin registry in the future!

The development masterclass took participants through some basic and some lesser-known features of Dyalog’s Integrated Development Environments. Some newer APLers said that they learned a lot, and even more experienced users said they had benefitted and would take the materials back to their organisations to share with others.

Adám Brudzewsky and Stefan Kruger host a masterclass on the features of the IDEs

Conversation over dinner included catching up with long-time friends and making new ones, stories of onboarding new APLers (some of them in attendance), and hearing about personal and professional achievements of the last few years. There’s excitement in the air this evening, and we can’t wait to start a fantastic few days of presentations tomorrow!

Follow