Dyalog ’22 Day 3: Automation, architecture and performance

Posted on October 11, 2022 by Dyalog

Links to recordings from this day of the user meeting are at the bottom of this post.

Automation, architecture and performance were throughlines of the second day of presentations. Lars Stampe Villadsen from SimCorp A/S provided some advice on how to write tests and gave a live demonstration of a small testing framework together with continuous integration tools which run the test suite every time a change is committed to the project – useful for when we forget to run some tests locally.

Lars Stampe Villadsen talks about testing.

Norbert Jurkiewicz presented his “10,000 ft overview” of automation processes used by The Carlisle Group and described some of the complexities of building distributables according to varying requirements for different customers. He discussed some of the cost considerations to be made when using Amazon Web Services (or any cloud computing service) to leverage lots of computing power to do builds quickly.

Norbert Jurkiewicz discusses automation architecture.

This was followed by Michael Baas with a similar issue viewed from a different angle. He talked about using the ]DTest testing framework to produce code coverage reports – showing which lines of code had actually been executed when running a test suite – and also automating testing across different platforms and versions of the Dyalog interpreter.

Michael Baas shows us the ]DTest framework

Changing tack, we heard the story of getting to grips with semi-global variables in a multithreaded application. Elena Pavarotti of SimCorp Italiana talked about her experience trying to imagine clearly what a complex system is doing so that it is easier to reliably refactor and adapt it to talk to new external systems.

Elena Pavarotti on managing complexity in SimCorp Sofia.

Rodrigo Girão Serrão then told us how himself and Aaron Hsu had implemented a U-Net Convolutional Neural Network from scratch in APL. The comparisons between their implementation and industry standard libraries was very interesting, and then seeing the mapping from diagrams to code showed us how the APL becomes a natural way to express the data flow involved in the system.

Rodrigo Girão Serrão walks us through the U-net CNN in APL.

Delving deeper into more intellectual musings, Justin Dowdy of Semantic Arts – also known for his work on the April APL to Lisp compiler and the May bridge between Dyalog and Clojure – drew some interesting analogies between the Resource Description Framework, used for representing data relationships as ontologies, and points raised in Iverson’s paper Notation as a Tool of Thought.

Justin Dowdy compares Notation as a Tool of Thought and concepts of Data Semantics.

Juuso Haavisto is a DPhil student at the University of Oxford. He brought up some hot topics in computer science academia – static analysis, rank polymorphism and scheduling for multi-core systems – that he believes can be tackled effectively if we can learn to make the computer think a bit more like an APLer.

Juuso Haavisto with three hot topics in computer science academia.

The theme of performance continued when Vali-Matti Jantunen from Statistics Finland compared the performance of some short APL phrases across several versions of Dyalog. Of course, this can be important in an application like PxEdit which processes thousands of text files; a fraction of a second difference processing a single file can lead to several minutes across a job.

Vali-Matti Jantunen on Dyalog performance across versions.

We hope Veli-Matti will forgive us for the performance regressions found in v18.2 (although not slower than v17.1). However, for us it was no surprise, and next Karta Kooner reflected on some of the assumptions made when changes, intended to improve performance, had been implemented over the lifetime of Dyalog APL. He is looking for volunteers to run a version of the interpreter which can gather usage statistics, so please get in touch if you can do this.

Karta Kooner analyses performance in the interpreter.

Presenting the final features of his special Dyalog ’22 Conference Edition of the interpreter, John Daintree talked about the state of asynchronous programming in the interpreter now. Eventually he showed us an interface, in the from of an ⎕AWAIT system function, to unify Spawn (F&⍵) threads, .NET Tasks and Futures and Isolates.

John Daintree with some ideas about asynchronous programming.

Finally, Aaron Hsu’s Co-dfns report showed a new public API for the parser which could be useful for static analysis of existing code and better error reporting within co-dfns. The new code generator written in APL opens the door to targeting more platforms more easily in the future.

Aaron Hsu presents an update on co-dfns.

Today’s presentations (links to recordings will be added as they become available):

U06: From “I Developed and Tested It” to “I Developed, and My CI Tested It” – Lars Stampe Villadsen
U07: Automating Application Builds with AWS – Norbert Jurkiewicz
D07: Test Your Code! – Michael Baas
U08: Semi-Globals and Multi-Threading are Like Chalk and Cheese – Elena Paviotti
D08: Implementing the Convolutional Neural Network U-Net in APL – Rodrigo Girão Serrão
U09: APL on the Side – Justin Dowdy
U10: Scheduling Array Operations – Juuso Haavisto
U11: Performance of Dyalog APL – A Historical Perspective – Veli-Matti Jantunen
D09: Performance Improvements in Set Operations – Karta Kooner
D10: 2022 Conference Edition Part 3a – Future(s) – John Daintree
D10: 2022 Conference Edition Part 3b – Future(s) – John Daintree
D11: Report on Co-dfns – Aaron Hsu

Dyalog ’22 Day 2: Welcome Back to User Meetings

Posted on October 10, 2022 by Dyalog

Links to recordings from this day of the user meeting are at the bottom of this post.

We were welcomed to the 2nd day of the user meeting by managing director (CEO) Gitte Christensen. Everybody could appreciate the emotion and sentiment when she noted the distinct “APL Hum” apparent in the atmosphere at this year’s event.

Managing Director Gitte Christensen welcomes us to the user meeting.

In some exciting news, Stine Kromberg announced that Dyalog’s board of directors have approved an APL fund for education and science. She also made a call to action for members of the APL community to present their stories to the wider world.

Stine Kromberg announces the APL fund

Morten Kromberg’s road map provided an overview of the multitude of projects and requirements Dyalog is to address in the upcoming years. From the need for new APLers to be recruited to replace those career programmers leaving the fold, to developing tools and training resources that can help all users navigate the modern, complex world of interconnected software development and making the user experience more pleasant and consistent across different platforms.

Morten Kromberg presents the roadmap of activity at Dyalog.

Morten later returned to speak about the status of project and package managers for Dyalog. While Carlisle Group’s Dado is an excellent recommendation for those who can adopt its tools and workflow, Dyalog Ltd has employed the help of APLTeam to develop the more agnostic Tatin package manager and Cider project manager, with the hopes of providing a more flexible and extensible set of tools to meet a wider range of needs.

In a double bill of talks from John Daintree, we were shown a special 2022 Conference Edition of the Dyalog interpreter which has features stemming from both internal discussions and projects and our users’ requests and ideas. He reiterated some of the complications of input and output in Dyalog, but showed how the conference edition can more intelligently handle output from the various sources – for example error messages, quad output ⎕←message and output from system commands.

In his second talk, he demonstrated a system for piece-by-piece debugging of inline APL code and functions. We think you will agree that being able to step through the evaluation of a single line of APL has massive implications for teaching, writing and understanding APL.

John Daintree demonstrates inline “token by token” debugging.

Charles Brenner from DNA-View took us on a journey of his discovery of a method of performing numerical integrations in high dimensional spaces – using a kind of Riemann summation (analogous to the rectangle or trapezium methods) using simplexes (multidimensional generalisation of triangle and tetrahedrons).

Charles Brenner talks about numerical integration

Gilgamesh Athoraya presented a recent project by Tiamatica AB to migrate a process planning system from mainframe APL2 to Dyalog in the cloud. This large undertaking began with creating a bridge from Dyalog to APL2, and the challenge continues as code must be migrate to Dyalog while being able to fold in updates made by the APL2 team still maintaining the current software.

Gilgamesh Athoraya talks about migrating APL2 mainframe to Dyalog in the cloud.

Another APL2 conversion, Mark Wolfson of BIG provided some fascinating insights into the world of Jewelers in North America. Many of these business have managed to resist the trend of becoming parts of large conglomerates and have stayed as small, often family owned businesses. This has meant that they usually don’t have sophisticated software for managing their sales and inventories. Mark’s system uses APL to ingest data from an incredible variety of formats and provide insights to help these businesses thrive.

Mark Wolfson gives insight into the world of North American Jewellery retailers.

Unfortunately, Kimmo Linna from Finnair could not join us in person due to his obligations as a pilot. Fortunately for us, his time away from the cockpit during the pandemic led to him developing useful tools and workflows that he was able to show us remotely via video feed. He is using DuckDB to store data and Vega-lite to present it visually, all being driven from Dyalog Jupyter notebooks and connected using bridges that he has published and made available for free on GitHub.

Kimmo Linna presents his Dyalog/DuckDB/Vega-lite/Jupyter workflow

To finish the day, new recruit on the development team Peter Mikkelsen talked about a personal project from before he joined Dyalog: an APL implementation for the Plan9 research operating system. As he explained, it may be lacking performance, documentation and even users, but we could see from his approach and language design choices that he is someone who can bring great ideas to the interpreter development team.

Peter Mikkelsen shows message passing in APL9

Today’s presentations (links to recordings will be added as they become available):

Welcome to Dyalog ’22 – Gitte Christensen and Stine Kromberg
The Road Ahead – Morten Kromberg
D03: 2022 Conference Edition Part 1 – Past – John Daintree
U01: One-Time Pure Mathematician Corrupted by Exposure to APL Loses Moral Compass and Discovers, after Several Mis‑Steps, a Useful Numerical Integration Method – Charles Brenner
U02: Lift-Off from APL2 Mainframe to Dyalog in the Cloud – Gilgamesh Athoraya
U03: Rebuilding a Production APL Environment using Dyalog – Mark Wolfson
D04: 2022 Conference Edition Part 2 – Present – John Daintree
D05: The P words…Projects and Packages – Morten Kromberg
U04: A Modern APL Workbench – Kimmo Linna
U05: An APL to Vega-lite Interface – Kimmo Linna
D06: APL9 from Outer Space – Peter Mikkelsen

Dyalog ’22 Day 1: Welcome to Sunny Olhão!

Posted on October 9, 2022 by Dyalog

Arriving to meet the warm sea air is already a refreshing change as this year’s long, hot summer started to close in the weeks leading up to this year’s user meeting. Of course, for some with less far to travel, the warmth is a familiar comfort.

Apartments at the Real Marina in Olhão, Portugal

As delegates arrived on Sunday night we were treated to a lovely surprise – a birthday cake to celebrate Tony Corso’s birthday! Happy Birthday, Tony!

Today we kicked off the user meeting with workshops, giving delegates hands-on experience with a range of Dyalog offerings for application development.

Attendees of Rich Park and Rodrigo Girão Serrão’s language workshops explored different ways to express ideas in APL using both tried and true idioms in the morning and newer language features in the afternoon – they compared expressiveness and performance implications in difference circumstances.

Morten and Josh got participants up to speed with using and maintaining code which lives outside of your workspace in text files, while Richard Smith and Bjørn Christensen showed how to manage and interact with data from outside of the workspace.

Brian Becker guided delegates into the world of SAAS (Software As A Service). It is encouraging to see how straightforward having APL code talk to the outside world can be. As usual, however, the complexities reveal themselves as you delve deeper in to specific use cases and circumstances.

Morten and Josh show how to store source code as text files using Link

Without a doubt, getting to meet our users face to face has already proved to be the greatest enjoyment of the meeting so far. Some familiar faces long since last seen, and others only over a screen. Discussing ideas face to face still retains a value impossible to quantify.

Overall a fantastic beginning to the week and we cannot wait to experience the rest of this week!

APL Seeds ’22: Tuesday 29 March

Posted on April 26, 2022 by Dyalog

On Tuesday 29 March we hosted APL Seeds ’22, the second annual online event for new and prospective users of APL (although everyone was welcome). Once again we were delighted to see that the majority of registrants had little to no APL experience; it feels like we get the chance to offer that same sense of discovery we felt when first learning about the language.

APL Seeds ’22 began with an introduction from Dyalog Ltd’s managing director, Gitte Christensen, in which she described her experiences of seeing APL enable people who had real problems to solve and showed some of what Dyalog provides in terms of the tools and interfaces that people might expect from a modern software development stack. Gitte explained the new Basic Licence, which is another step forward in Dyalog Ltd’s aim to bring APL to a wider audience. The Basic Licence allows non-commercial distribution of Dyalog along with APL-based solutions under the terms of the Royalty-Based Run-Time Licence, which will apply as the default run-time licence (see Prices and Licences for more information on Basic Licences). She also described some customer use cases, some of which might surprise newcomers to this language. Rich Park finished the introduction by pointing out where you can find more APL content, especially if you’re just getting started. For example, our tips for beginners includes things that might not be obvious when you first start the interpreter or read introductory books. The video description for the recording of this presentation contains many useful links!

APL is an executable notation and tool of thought which enables people with good ideas to bring them to life with computers

Gitte opens the event with her thoughts on “What is APL?”, including the short expression 1 2 3 + 4 5 6 that started the journey which eventually brought her to Dyalog.

Rich then presented a basic introduction to APL, showing the benefits of a symbolic notation for programming as well as demonstrating how to put together simple building blocks to build a function. After introducing the basic syntax, right-to-left precedence, and the generality of APL operators, along with a handful of symbols including the famous outer product (⍺∘.F⍵) and array indexing (⍺[⍵]), he walked through constructing a function that visualised the probability distribution for sums of rolling two N-sided dice.

A screenshot of an APL function to do some simple statistical computation and display the output using characters in the APL REPL

The Dist function uses just a handful of APL constructs to create a visualisation of a simple statistical distribution.

Stefan Kruger, author of the online book Learning APL, took us on an exploration of bioinformatics problems – a popular topic for the annual APL Problem Solving Competition. He described what a “k-mer” is (a chunk of DNA of a particular length), and compared different techniques for cutting up a text vector to isolate them from a DNA string, including a windowed-reduction (⍺F/⍵) and its generalised cousin, the stencil operator ((⍺⍺⌺⍵⍵)⍵), and our old friends the outer product and array indexing. Finally, Stefan looked at three approaches to doing some simple statistics in the Rosalind challenge “Computing GC Content“. To our delight, a new user who was in attendance commented that they learned new expressions and idioms that they had not seen before.

Stefan compares 3 functions to split a string into lengh-4 substrings

Andrew Sengul presented a more involved example, April. The April APL Compiler is a new entry in the APL field, compiling a subset of APL into the Common Lisp language and allowing APL functions to easily be used within Common Lisp programs. Andrew gave a concise history comparing Lisp and APL. He then gave a small introduction to using Lisp to write a “macro” (code that generates other code) before giving a glimpse into the implementation, architecture and design of APL in Common Lisp, as well as code that combines APL and Lisp to create visualisations. He showed a visualisation that used APL to both run Conway’s Game of Life and apply convolution kernels to show the state of cells over time. We loved seeing how he has been using April in the visual art installation Bloxl (a collection of computer-controlled light-up blocks used at events for a stunning visual effect). Andrew concluded his presentation with a demonstration of April code implementing a “falling block game”, and a video of that game in action.

Andrew gives a comparative overview of the histories of the APL and Lisp programming languages

Finally, there was a live recording of an episode of Array Cast, a semi-weekly podcast about array languages. From the regular panel of presenters were self-proclaimed J enthusiast Bob Therriault, Kx Librarian Stephen Taylor, Dyalog tools developer and life-long APL programmer Adám Brudzewsky, and professional C++ developer and programming language fanboy Conor Hoekstra. They were joined by a very special panel of guests: speakers from the event Gitte Christensen, Rich Park, Stefan Kruger, and Andrew Sengul, and well-known APLers Aaron Hsu and Rodrigo Girão Serrão.

The discussion began with the common beginners’ questions of keyboards, typing APL and whether you really need stickers, keycaps or a whole dedicated APL keyboard to use APL. On the topic of actually learning APL, there were mentions of even more books and other resources, including YouTube channels run by some of the podcast panellists and guests. All relevant links are included in the show notes for the episode, and you can listen to the episode on arraycast.com.

Taking a more technical turn, there was discussion of the balance between code clarity and performance. Should code be clear unless absolutely performance critical? Or is it possible to have both, where the clearer encodings and approaches are also the fastest? The episoded was capped off nicely, in response to a question from the audience, with Gitte offering her perspective on how APL can help a data analyst or engineer.

In the informal meet-up after the talks, Andrew configured a simple interface to the aforementioned “falling block puzzle game”, in which participants could control the game and see their moves played out on a Bloxl wall streamed in real (if a bit delayed) time. The players tried their best but were ultimately thwarted by the interface of clicking buttons using Zoom shared controls!

To those who attended, we hope you found the event enjoyable. Relevant materials have been uploaded to the APL Seeds ’22 webpage, including links to recordings of the presentations on dyalog.tv.

D01: Welcome to APL Seeds ’22 (31 minutes)
D02: APL – Easy to Learn, Worth Mastering (27 minutes)
U01: What’s a k-mer? (37 minutes)
U02: April: An APL Compiling To Common Lisp (39 minutes)
U03: The Array Cast (live podcast recording) (63 minutes)

Not All Heroes Wear Capes

Posted on October 27, 2021 by Dyalog

On Friday 15 October, the SERV S&L was presented with The Queen’s Award for Voluntary Service by the Lord Lieutenant of Surrey. Geoff Streeter was one of 20 volunteer SERV S&L members who attended.

What Is SERV S&L?

SERV S&L (Service by Emergency Rider Volunteers for Surrey and London) are a charity organisation, made up entirely of volunteers, comprising motorbike riders, car drivers, controllers, and fundraisers. They transport blood products, urgent samples, medical supplies, and donated breast milk to hospitals and milk banks across Surrey & London, as well as carrying out a daily delivery of blood to the Air Ambulance service that covers Kent, Surrey, and Sussex. They support the regular delivery rounds that the NHSBT (National Health Service Blood and Transport) have in place; unlike the NHSBT, SERV S&L also operate throughout the night. All of this is provided free of charge to the NHS, releasing more money for patient care.

What Is The Queen’s Award for Voluntary Service?

QAVS (The Queen’s Award for Voluntary Service) celebrates the outstanding work of local volunteer groups across the UK. Created in 2002 for Queen Elizabeth II’s Golden Jubilee, QAVS awards shine a light on the fantastic work of voluntary groups. QAVS awards are the highest awards given to local voluntary groups in the U.K. (they are the equivalent of a personal MBE) and they are awarded for life.

Geoff’s Involvement with SERV S&L

A Personal Recollection

At the end of 1980, Paul McCann had a relation who could not get an urgent sample transported to the testing lab until the next morning. He was frustrated by this and organised a meeting to see what could be done, the result of which was that a group of advanced motor cycle trainers from a (now defunct) group called Star Rider decided to try to run a delivery service for blood/samples at night. I was not at that meeting but I heard about it from a fellow member of the Laverda Owners group; I made it to the second meeting (on 8 December 1980) and have been involved ever since. We obtained a room with a couple of bunks in a wooden building owned by MEFAS (Malden Emergency First Aid Society) and a telephone line, and started operating in early 1981.

The main distribution point for blood is located in Tooting and serves London, Kent, Surrey and Sussex (we have partner organisations in Kent, Sussex and Wessex). We do a main nightly run with typically 6 to 10 boxes down to an arranged change point for Kent and Sussex. We also partner with similar organisations across the U.K., and have occasional relay runs, for example, from Edinburgh to central London (I think that’s the longest that we’ve been involved in). More common are runs from Bristol. We typically shift 20 boxes a night and samples in the other direction and have about 8 riders/drivers on shift every night.

Financially, we get support from some Masonic Lodges and business groups. They prefer to buy bikes for us, and Citroen have given us a car (DS3) on permanent loan. We are in the process of acquiring/refurbishing a scout facility in Sutton to provide a base for the bikes/cars/van as well as for volunteers who live on the periphery of the area. We also raise funds by box waving outside supermarkets, garden centres, Brooklands, Waterloo Station, etc.

I started with the group as a biker, and used my Laverda 750, Laverda 1200 and Honda 650 Turbo to deliver blood and samples from 1981 until 1990, when I switched to car deliveries (which I continued to do until last year). I also acted as Treasurer from 2006 until 2010. I have been one of the controllers right from the start [Ed.: Controllers orchestrate the logistics of a shift; hospitals and partner groups place their orders and riders and drivers are dispatched as required – accurate scheduling and data logging are required to ensure efficient co-ordination and communication so that each run can be completed reliably], a role that has changed a lot over the last 40 years. In the early days controllers needed to be physically present with the one rider and the telephone. Then we moved to using pagers (but still needed to be present in the hut/sports centre) before everything changed with the advent of mobile phones – I now control from home. The expectation is that volunteers do one night a fortnight, but a shortage of volunteers relative to growing demand means that for a few years now I have been doing at least one shift a week.

Final Word

Congratulations Geoff, 40 years of volunteering for such a worthy cause is a fantastic achievement. All of us at Dyalog Ltd are really proud of your contribution.

To find out more about the amazing service provided by SERV S&L, including how to make a donation, visit https://servsl.org.uk/.

Welcome Karta Kooner

Posted on August 23, 2021 by Dyalog

Karta joined Dyalog in April, and is yet to meet anybody in person although he’s been told that this is not necessarily a bad thing! After completing his doctoral degree in theoretical physics, Karta stumbled upon Dyalog and APL entirely by happenstance. Being often captivated by things that look unfamiliar to him, and having an interest in most things, it was a code golf question that was answered in a strange, yet mathematical-looking language that took him to the profile of the poster, who happened to mention they were employed by Dyalog and currently hiring. He sent an email enquiring about the opportunity and, several remote interviews later, was happy to be hired as a C/C++ developer working on the interpreter.

Karta is one of the few members of the team that knew no APL whatsoever before joining and has been very impressed by Dyalog and APL thus far; he is very much looking forward to seeing how far the language can be taken, with an eye to further developing and potentially encouraging its use in academia and other technical fields of study.

In his spare time, Karta enjoys expanding his knowledge of both scientific and technical pursuits, and tinkering around with software and hardware systems, amongst his eclectic interests. When not found reading papers or learning an unfamiliar branch of mathematics, he will be caught thinking of a new engineering project to occupy his time, or stumbling through learning a new language, or maybe just delighting in the latest vixra paper.

The tool of thought for

software solutions

Contact Us

Author Archives: Dyalog