Announcing the Beta Programme for Dyalog APL Version 18.2


I am very pleased to be able to announce the start of the Beta testing of the next release of Dyalog APL! As explained in June, we decided to delay the release of version 18.1 in order to take a closer look at some of the optimisations that had been implemented in 18.0 (and were therefore also present in 18.1).

Our analysis of the optimised code concluded that, due to the nature of many of the new algorithms involving modern vector instructions and code generated from templates, we need significantly more time to create tests that will cover absolutely all the different cases that have been implemented. As a result, we do not feel that it is in the best interests of our users to release v18.1 in its current form.

The good news is that the features of modern source code management systems, combined with our collection of regression tests, have allowed us to create a new version of Dyalog APL that contains all of the new features added to versions 18.0 and 18.1. This includes bug fixes made during these two development cycles, but not the optimisations that make us feel uncomfortable. To differentiate this version from the existing version 18.1, we have decided to call the new version 18.2.

Everyone who was signed up for the v18.1 beta programme should now be able to download v18.2 beta. If you are not signed up as a beta tester already but would like to help us with testing, please get in touch. Under Microsoft Windows, testing should be significantly simpler this year, as we have started producing Microsoft Patch files (MSP) as the delivery mechanism for updates – something I have personally been looking forward to since before I joined Dyalog Ltd!

Version 18.2 Performance

One unavoidable consequence of the above is that the performance of v18.2 is closer to that of v17.1 than to v18.0. Our own tests show that we have not been pushed ALL the way back to “square one”: v18.2 appears to be slightly faster than version 17.1. Once v18.2 has been delivered we will work on carefully re-implementing the most valuable optimisations that have been removed. We welcome your input on which primitives you think we should speed up first, so please participate in testing v18.2 and let us know what you think we should prioritise as we start work on version 19.0!

Recommendations regarding Version 18.0

Over the summer, only one additional defect related to performance optimisation was discovered and fixed. We are not currently aware of any outstanding defects in v18.0 caused by recent optimisations. Dyalog Ltd is committed to providing support for version 18.0 until the arrival of the 3rd release following it, in accordance with normal policy.

However, if you have not yet upgraded to v18.0, Dyalog Ltd strongly recommends remaining on your current version and moving directly to v18.2 when it is released. If all goes well, this will happen at the end of 2021 or very early in 2022. If you are already using v18.0, then we recommend that you make plans to start evaluating v18.2 and moving to it as soon as possible.

Conclusion – and Apology

We are painfully aware that the defects found in v18.0 and the resulting uncertainty have seriously inconvenienced some of our users, and I apologise for this. The root cause is the growth and rejuvenation of the Dyalog development team. Our original processes for quality assurance relied on years of tacit knowledge; when enthusiastic new team members break significant new ground, more explicit planning and QA processes are required to make sure that new approaches are safe and stable.

When we resume work on optimisations following the release of v18.2, this will be done according to new guidelines that require the process to begin with a careful risk/benefit analysis of any enhancement to primitive functions. We will do everything that we can to move forward in a way that will allow us all to eventually look back on the events of 2021 as a significant step towards a more capable and reliable development organisation and product.

After all, in another two years it will be time to celebrate 40 years of Dyalog APL!

Essays on APL Since 1978

In June, Mikhail Barash and Anya Helene Bagge published a collection of essays written by the students of the seminar course INF328B on History of Programming Languages that was given at the University of Bergen (Norway) in the Spring term 2021. As an author, I wish to start by thanking Mikhail and Anya and the students for this initiative! It is inspiring to those of us who have been in this game for a while to see young computer scientists who are interested in language design studying history – it definitely makes us feel that the effort put in to writing the paper was worthwhile!

Each essay summarised one paper from the HOPL IV conference (part of ACM SIGPLAN’s PLDI 2021). Two students wrote essays on the paper “APL Since 1978”, by Roger Hui and myself. At the end of each essay, students were encouraged to pose questions to the authors of the HOPL IV papers, and this blog post has essentially been written in response to those questions.

Karl Henrik Elg Barlinn asks:
“I wish to know if there are plans to try and popularize APL within the wider community of programming languages. I ask this because I see how it can be very useful for the mathematical community to write papers and be able to execute the notation.”

The APL community has always seen “evangelism” as an important activity, and that remains true to this day. It is a challenging task because the most successful APL users are neither software engineers nor mainstream mathematicians, but various types of domain experts who are able to apply mathematics and learned enough about programming to write high value applications with a lot of domain-specific content. Typical users have been actuaries and financial experts, operations researchers and planners.

Successful APL users are more likely to write papers at an Actuarial or Chemical Engineering conference, and typically lack the vocabulary and the insight into mainstream computer science to present APL to the “community of programming languages”. They also typically work in highly competitive industries and have little time or inclination (or permission) to publish their work.

The situation is improving: the growing interest in functional programming, and the general recognition by the software engineering community that there is value in combining different paradigms (as opposed to the bad old days when everyone thought that the world would soon standardise on C++ or Java), makes it much easier to interest the new generation in APL.

A new generation of APL users is emerging, who have more insight into Computer Science and are more able to bridge the gap. Examples of recent work include:

Dyalog Ltd recently introduced an event aimed at new users; recordings from the first of these meetings can be found at https://www.dyalog.com/apl-seeds-user-meetings/aplseeds21.htm.

In addition to the marketing efforts, Dyalog is working hard to add tooling that will open APL interpreters up to mainstream “devops” workflows, based on text-based source files. Historically, most big APL shops developed their own home-grown management systems (many of them pre-dating tools like SVN or Git by decades), and there has not been a lot of tooling shared by the community.

“Given unlimited influence, where do you wish to see APL be used? If your answer is everywhere, does that mean APL is fit to do everything? On the other hand if your answer is not everywhere, where is not fit to be used and why?”

Given unlimited influence, I would push APL as a tool for education. I think it could be a useful tool for teaching mathematics – and how to use it to solve problems on a computer – starting with children. The simplicity of APL’s syntax means that it is also a good tool for teaching people of all ages and at all levels of education to manipulate data without just feeding it to ready-made packages. Teaching fundamental algorithms in computer science classes is also a good place to use APL, although the CS establishment will probably question that since APL tries to do most things without loops or type declarations, and sort of “skips over” the lowest level of algorithms.

APL is a useful tool for modelling, prototyping and designing solutions to any kind of problem. Obviously, many domains already have tools specifically designed for common types of problems. For example, Mathematica and MatLab have built-in solutions for many different classes of mathematical or engineering problems, TensorFlow for artificial neural networks, and so on. However, when the time comes to perform a major revision or extension, or there are no pre-built solutions, APL will be a good choice for prototyping.

Although APL is perhaps most valuable during analysis and design, the “executable design” is often used in production because APL interpreters are efficient on the relatively dense data structures that results from array orientation and because the ability of domain experts to write code (and tests) eliminates many sources of errors and poor performance due to unnecessary abstractions.

APL is not always an appropriate choice for the final production system. For example, I would not use APL to implement a real-time system, at least not using existing APL interpreters, which will freeze up to do compactions every now and again. For mission-critical systems, the additional safety provided by strong typing or other mechanisms for verifying correctness, and using teams trained to focus on reliability rather than analytics, may have benefits. If the core algorithms are not array-oriented and require a lot of looping or recursion, an interpreter may not be the right solution, and APL compilers are not yet mature technologies.

Even when the final production system is rewritten in another language, the prototype can be useful as a verification system, especially because the implementation is likely to be radically different, and can therefore almost act as a proof of correctness.

For an example of how APL can add significant value even when it is not used in the final implementation, see Martin Janiczek’s presentation at APLSeeds21, on “How an APL Prototype Helped Designing a Service”: https://dyalog.tv/APLSeeds21/?v=qDl3obmOd58.

“Q: Do you think APL with its glyphs is more fit to be taught in school than J, as special equipment is no longer an issue with UTF being widely adopted?”

There are still some problems related to APL symbols, such as many applications rendering the symbol incorrectly as a followed by a slightly offset / unless a supporting font is used. I suspect that it may be a while before we have handwriting recognition for APL symbols, or support for APL in writing systems for the visually impaired, and so on. On the other hand, one of the benefits of APL (which also holds true for J) is that it is independent of any particular human language, without needing to be translated from English.

Roger Hui commented in the HOPL IV Slack channel: “I am guessing that if the ecosystem for Unicode was more developed at the time (1990) Iverson would have kept the APL glyphs.”

Sondre Nilsen comments:
“If I would have some feedback it’d be to include an “array programming languages for dummies” appendix that could be used to look up foreign concepts, words and phrases that unfamiliar aspiring APL developers may not know.”

Hopefully the APL Wiki (https://apl.wiki) will be a useful resource, along with the (APLCart https://aplcart.info) and the evolving digital version of Mastering Dyalog APL. Please take a look at let us know if you feel more is required!


Roger and Morten’s HOPL IV paper “APL Since 1978”:

Thank You Ian Sharp

On July 16th, one of the most influential founders of what we today refer to as the “array language community” died peacefully, a few months after being diagnosed with lung cancer (link: Toronto Globe and Mail).

Ian Patrick Sharp

In 1964, Ian Patrick Sharp formed I.P.Sharp Associates (IPSA), together with six colleagues who were made redundant when Ferranti-Packard closed its computer division in Toronto, Canada. As Ian explains in a wonderful interview that was recorded in 1984 (link: Snake Island website), he was approached by people who wanted to recruit the whole team. Instead, he decided to form a company, since the team obviously had significant value.

The company was involved in the first APL implementation at IBM (APL\360). Subsequently, IBM allowed them to modify and enhance the system, and built a timesharing service that became known as SHARP APL. Roger Moore was a co-founder of IPSA and, in addition to being responsible for the supervisor that made SHARP APL a superior timesharing system, Roger was the chief architect of IPSANET, one of the worlds first packet switched networks.

In the late 1970s the combination of APL and IPSANET was revolutionary, and IPSA quickly attracted business from global corporate clients who used SHARP APL for e-mail, reporting and analytics, and a rapidly-growing collection of financial timeseries data – all completely new technologies at the time. In particular, the transmission of data over telephone lines changed the world. Ian had many absurd encounters with telecom monopolies who tried to protect old business models or profit from the new technology (link: archive.org).

A Stylised Map of the I.P.Sharp Associates APL Time-Sharing Network

Ian’s management style perfectly matched – and drove – the revolutionary technologies. As Ian explains so eloquently and humorously in the interview, IPSA recruited talented people without necessarily having specific tasks in mind. Ian set the tone and direction and then let people get on with it, moving around in the background to get a sense of how things were going and making adjustments without ever making a fuss. IPSA was a fantastic place to work and attracted a wonderfully diverse (in the most modern sense of the word) collection of smart people who developed revolutionary tools, helped a lot of customers, had a lot of fun, and made money.

Ultimately, IPSA was creative, problem-oriented and customer-driven to the extent that it failed to respond to fundamental changes in the market in time. At the end of the 1980s the timesharing revenues suddenly faded, and the company was acquired by Reuters for its timeseries databases and more or less disappeared overnight. However, IPSA had acted as a fantastic breeding ground for technology and talent for a quarter century, and there are hundreds of people who fondly and gratefully remember Ian for the way that he allowed them all to grow.

I don’t think it is a coincidence that so many of the active array language organisations have key players who were once IPSA employees (some of them appearing in more than one place thanks to relationships forged a very long time ago 😊).

  • Jsoftware: Roger Hui, Eric Iverson, Chris Burke, Ken Iverson
  • Kx: Arthur Whitney, Simon Garland, Stephen Taylor, Chris Burke
  • Dyalog: Gitte Christensen, Morten Kromberg, Roger Hui, Brian Becker, Dan Baronet
  • Snake Island Research: Robert Bernecky

As always, Roger has collected anecdotes about IPSA, Ian and other people who worked there, which you can find on jsoftware.com/papers/SharpQA.htm.

My Own IPSA Story

In 1978, my dad was moving out of an apartment in Oslo. At the same time, XEROX insisted that IPSA open an office in Oslo to support their international business, and several Canadians arrived there. I helped move some furniture and, sensing a keen interest and real excitement in programming, the IPSA Oslo team offered me a free account to play with APL timesharing, if I was interested. I effectively became a piece of furniture in the IPSA office after school, and had keys to the office so I could come and go as I pleased. After a year or so, they started throwing me bits of real work to do and paying me for my time. I think I was 17 at the time.

In addition to working as an APL consultant and tool builder, one of the things I did in my spare time was to write a tool for myself that would compare the entire contents of the e-mail directory with its state at the end of the previous week. Since IPSA was 100% managed by e-mail groups, this allowed me to know instantly when a new office was opened, a significant new project was started, and, of course, when new employees joined the company. By using this technique of harvesting e-mail addresses and sending unsolicited e-mail when an interesting project or person joined, I found my future partner both at home and the office – Gitte, the current CEO of Dyalog Ltd – only about 500km away in the IPSA Copenhagen branch.

I spent about a decade at IPSA and, after its sudden disappearance, Gitte and I have been trying to recreate the IPSA atmosphere in every team that we have been a member of. In a very real sense, I owe not only my career but almost everything of value about my life to Ian Sharp and the warm and welcoming company that he created.

Thank You, Ian!

The APL Orchard

If you have not already been there, I highly recommend visiting the APL Orchard (apl.chat). Adám Brudzewsky originally started this chat room on Stack Exchange to teach and answer questions, including many successful introductory sessions to people who wanted to learn APL. Since then, it has become an extremely active discussion forum, with a very wide range of interesting conversations, from regularly helping more newcomers get started with APL to theoretical discussions about the design of future array languages that will improve APL and perhaps compete against it some day.

Interested in APL?

One of Adám’s infamous invitations to a personalised APL introduction

Several of the most active participants have written their own APL implementations and some are working on array languages that are quite different from APL. It is a lively crowd, generating lots of thought-provoking discussions that feel a lot like the big arguments that I remember witnessing between Iverson, Benkard, Bernecky, Falkoff, Trenchard More and others at APL conferences back in the days when the APL language was still being born.

The participants include a mix of current, past and future employees of Dyalog Ltd, and it is a source of much useful inspiration for future work on our product and healthy challenges to conventional wisdom, so I try to spend some time there every day.

APL Orchard Stats

APL Orchard message statistics. The bottom contains the all-time total number of participants and messages.

To the New APLers

The Orchard is sometimes a challenging environment for a Dyalog CTO. Many of the bright young minds with computer science backgrounds that arrive here are quick to latch on to the fact that many things about APL go against the grain of what they were taught at university, and that a few of the design decisions made during the 55 years since the first APL implementation were questionable and could do with rationalisation. There is a constant barrage of complaints that we are not working hard to “fix” the things that they consider to be wrong about APL. This is, of course, how things should be when youthful enthusiasm meets us “fossils”; I fully understand many of the points made and actually agree with quite a few of them.

It is sometimes hard work having to repeatedly defend why Dyalog does not do things like just fix index origin at 0 and give the customers a few years to refactor their code. The implication is often that we are incompetent, and occasionally there is the insinuation that we are driven by questionable commercial motives like trying to get rich by locking the customers in and then doing a minimum of work.

Evolution, not Revolution

In my opinion, a large part of the value of our product rests on the fact that our customers can do things like read new legislation that affects the way their code needs to compute something, write code to deal with it before lunch, and then expect the code to keep working without changes until the law changes again, which might be in four years (depending on which country you live in), or in a few decades, after the current programmers have retired. Meanwhile, the code might need to move from the mainframe via UNIX and DOS to Microsoft Windows or macOS and then on to Linux in the cloud, and whatever comes after that.

APLers actually do enjoy refactoring code *if* it is in order to meet new challenges in the markets that they serve with products based on APL or compute better results. Our duty is to make this easy for them. On the other hand, there is no business value in refactoring code because we decided to change the way the language works, due to theoretical considerations – and it is our duty to protect them from that.

Breaking changes to Dyalog APL are quite simply not an option, at least not if users have to take immediate action. When we accidentally make a breaking change despite our best efforts not to, there is much wailing and gnashing of teeth, and we have to drop everything to fix it. That doesn’t mean we can’t make significant changes to how things work, but we must take an evolutionary approach, where the old way continues to work until everyone who needs it has moved on, possibly controlled by (sometimes undocumented) switches, like the one that forces APL to continue to give a DOMAIN ERROR on ¯1*0.5 to protect legacy code that relies on trapping the error in certain financial calculations.

Over the past decade, we have been able to move almost all of our user base from a 32-bit product with a fixed “alphabet” of 256 characters to using 64-bit APL with full Unicode support, in many cases without requiring any changes to application code. The stragglers are mostly major clients with support contracts that justify the additional cost of continuing to support the “legacy” version of the product until they are eventually able to move on. Once the Raspberry Pi stabilises as a 64-bit platform, we may finally be able to completely retire 32-bit Dyalog APL. I estimate that the “Classic” (non-Unicode) version probably needs to exist for another 5-10 years.

The next big challenges for us are to ensure that code can be moved to macOS and Linux (and “the cloud”) with a minimum of changes, and that both new and existing users have ways to integrate both new and legacy APL code nicely with modern source code management systems and continuous integration pipelines.

Existing Customers First

Our first priority at Dyalog is to support our existing clients and make sure that they remain competitive in their respective marketplaces.

Some contributors to the APL Orchard have suggested that this prevents us from being able to attract new users, which they believe can only happen if we quickly fix some of the “warts” in the language, or add some even more powerful language constructs.

First of all: not a single person who I consider to be a real prospect for writing substantial new application code in APL has brought up a single one of these issues as a reason why they might not want to adopt APL.

Secondly: if we do not first ensure our own financial stability, we will not have the resources to perform evolution, let alone revolution. For example, our investment in the creation of the APL Orchard, the APL Wiki, and the creation of new training materials for potential new users of APL, all depend on this.

As they say in the pre-flight briefing: put on your own oxygen mask before helping others.

Third: it is my very strong conviction that making breaking changes would violate the trust that exists between Dyalog and its customers, and I would personally consider it to be unethical. This may sound like a radical position, but it is one that I would expect to resonate with the younger generation. My personal motivation for working on software stems from witnessing the value that is created when people use software that I have helped create. Once someone starts using “my” product (and they continue to indicate that it is important to them by paying for support), I have an obligation to protect them from harm, if it is in my power.

Conclusion

APL is Fun
Old timers: If you are looking for some interesting discussions about current and future array languages, to help some newcomers, and maybe learn some new tricks yourself, I suggest that you head over to the APL Orchard and dive in, it is a lot of fun! You might want to check out apl.wiki/APL_Orchard, which has some helpful hints and tips, before you start.

New folks: I hope that this explanation of some of the philosophy that drives our work at Dyalog helps to explain some of the decisions that we make and that you find to be puzzling. I fully recognise that it is hard to imagine that a software company might have customer relationships that started before you were born 😊.

There are a couple of other topics that I should possibly also explain my views on, such as the dilemma of whether APL is a Notation or a Programming Language, and why Dyalog APL is not open source, but this post is very long already. Depending on the response I get to this post, I may return to that in the future.

Dyalog ’20 – Recordings Now Available

We are happy to announce that the full set of recordings from Dyalog ’20 online is now available. So if you missed the all or any of the talks, or would like to revisit one of the presentations, head over to https://dyalog.tv/Dyalog20!

It was disappointing not to be enjoying Portuguese food and drink with you all in Olhão. On the other hand, it was wonderful to be able to share our plans and user stories with so many people who would not normally be able to travel to one of our user meetings. According to the statistics, we had about twice the usual number of attendees, and Dyalog ’20 may have been the largest gathering of APL users in the last quarter century!

We learned that we need to invest in better microphones and find better solutions for chat both during and between the presentations, but in general we feel that the online format worked so well that we are making plans to run similar events in the future, even if international travel restrictions should ease and we are able to meet many of you face to face in Portugal this coming October. We are still thinking about the details, but it is likely that we will host an online meeting each spring with a focus on new users of Dyalog APL, while the autumn (fall) meeting will continue to provide experienced users with the usual “deep dive”.

We are also planning to offer workshops and other training sessions at other times of the year, and continue the regular series of webinars. Travel restrictions are helping to accelerate our plans to provide a steadily increasing quantity of online material. If there is sufficient interest, I am willing to expand my talk on Docker containers into a half-day “Bring Your Own Application” workshop early in 2021. If you would like to attend this workshop, or you have ideas for other topics for webinars, workshops or talks at future user meetings, please write to usermeeting@dyalog.com and tell us about it!

Warming up to Dyalog ’20 Online with the last Recording from Dyalog ’19!

Three weeks from today, Gitte Christensen and I will be opening the 2020 Dyalog user meeting. Like so many similar gatherings, we are moving online; this year’s meeting will consist of two 4-hour sessions on Monday 9th and Tuesday 10th November, running from 14:00 to 18:00 UTC. Our hope (so far confirmed by advance registrations) is that this format will allow many more people to attend than a normal meeting, which requires you to set a full week aside and spend money on travel. We hope that the timing will allow attendance from a large part of the globe, and of course it is our intention to make recordings available afterwards for those unable to attend live.

I do subscribe to the proverb about all clouds having silver linings: although all the user meetings that I usually travel to attend have been cancelled or postponed, and I have missed meeting many of you face-to-face this year, it is important to note the unexpected side-effect; there is significantly more APL content being generated and made generally available than ever before! I hope that you have all noticed that Dyalog and the British APL Association have been holding online sessions every two weeks since the spring! A good place to keep track of these events is our event calendar at https://www.dyalog.com/dates-for-your-diary.htm.

Two four-hour sessions is obviously a lot less than the usual 3.5 days plus workshops. You should be able to find a good deal of the “missing” material, such as Adam Brudzewsky’s five-part series on features of Dyalog Version 18.0 (and my own introduction to the new release) at https://dyalog.tv/Webinar, where you can watch it at your leisure.

During Dyalog ’20 we will focus on giving you updates on the most important activities that the team behind Dyalog APL is currently working on. Also, although you may have to wait for Dyalog ’21 in Portugal to meet the winner of the APL Problem Solving Competition in person, Andrii Makukha from the University of Hong Kong will be giving his acceptance presentation online on the Tuesday. Chris and Michael Hughes will give us an update on a tool called qWC, which simplifies the transformation of existing Windows applications into web apps.

Calculating estimates for the paths of debris from M/S Irma.

Calculating estimates for the paths of debris from M/S Irma.

At Dyalog ’19, Tomas Gustafsson, author of the stunning Stormwind boat/ship simulator (real time physics engine implemented in Dyalog APL) told the story of the successful search for the M/S Irma, which was was lost in a sudden autumn storm between Finland and Sweden 50 years ago. Because a Finnish TV channel was producing a programme about the Irma, Tomas asked us not to publish the recording of his presentation last year. The embargo has finally been lifted, and we are now able to present the final recording from Dyalog ’19. In Tomas’ Dyalog ’19 video, you can hear Tomas tell the story of how, despite massive search efforts, said to be the biggest ever in Finnish history, the Irma accident became a mystery. No distress signals were heard during that fatal autumn night, and Irma had chosen the weirdest routes. Only one body was ever found, days later, at an unexpected location. Wreckage from the ship was discovered in the archipelago at multiple contradictory spots. Irma was found in May 2019 by a group of enthusiasts (including Tomas). The reverse drifting patterns calculated using Dyalog played a crucial role in the success of the search. Tomas will be back at Dyalog ’20 to entertain us with the story of his latest adventure: the search for a 500-year-old wreck, the Hanseatic hulk Hanneke Vrome, which left Lübeck at the brink of winter in 1468, to avoid Danish pirates.

I look forward to welcoming you to Dyalog ’20 on November 9th.