News and Recommendations Regarding Dyalog Versions 17.1, 18.0 and 18.1

Background

Dyalog version 18.0 contained the largest number of optimised algorithms in the history of Dyalog APL. Unfortunately, since its release, we have found that our process for review and testing of optimisations was insufficient to cope with the quantity and nature of the optimisations. Many of the new algorithms had edge cases that escaped both the existing regression tests and the new tests that were written as part of development.

In August 2020, after the issues were reported following the release of Dyalog version 18.0, we issued a caution against the use of version 18.0 in production. We added significant additional testing during September and October, and subsequently lifted the caution in November. In April 2021, a user encountered an additional defect leading to incorrect results in a rare case of membership (), and in June a crash in an extremely rare case of “where” (monadic ). Since then, our own internal testing has found a defect in interval index (dyadic ).

Current Status

We have made patches available as defects have been detected and fixed, and “Issue 4” of version 18.0, made available to all users on 22 June 2021, includes fixes for all known issues with optimised code.

Download Dyalog version 18.0 Issue 4: Commercial users  Non-commercial users

At this time, we are executing two high priority projects, one aimed at increasing safety in the short term, the other at completely eliminating problematic optimisations within a few months:

  • We have further increased our internal testing of version 18.0, to verify the correctness of the optimisations that are still in the distributed product.
  • We are holding back the release of version 18.1, which was originally scheduled for release this month, while we remove most of the optimisations that went into version 18.0. We hope to release version 18.1 before the end of the 3rd Quarter of 2021, but will not let it out the door until we have complete confidence in this new version.

Recommendations

Dyalog Ltd recommends that users of Dyalog APL take the following action:

  • If you are using version 17.1 or earlier, plan to skip version 18.0 and upgrade directly to version 18.1 when it becomes available.
  • If you are using version 18.0, apply the latest patches or reinstall using Issue 4 when it becomes available. Continue to monitor our DSS e-mail broadcasts and apply patches quickly if we should find and fix any further issues that you think might impact your application. When version 18.1 becomes available, plan to upgrade to it at your earliest convenience.

Support for Versions 17.1 and 18.0

Considering this extraordinary situation, we have decided to extend support for version 17.1 for one additional release cycle. In other words, version 17.1 will be supported until we release the 4th subsequent version of Dyalog APL. Note that we are about to provide an updated version 17.1 installer (“Issue 3”) which collects all updates to version 17.1 since the original release.

Version 18.0 will be supported for the normal number of cycles, that is, until we release the 3rd following version. However, Dyalog Ltd recommends upgrading to version 18.1 as soon as it becomes available and you are able to schedule the upgrade.

Non-Commercial Versions

Given the nature of these defects, it is our intention to update our non-commercial distributions of version 18.0, although this process may lag behind the distribution of patches to clients paying for support by some weeks.

Conclusion

We apologise for the significant inconvenience that we know this is causing for some of you and thank you for your patience. We have made significant changes to our internal procedures regarding risk assessment, verification, and testing requirements for changes to existing primitive functions, to avoid a recurrence.

We do expect to re-apply those optimisations that we believe provide significant performance benefits to justify the risk of making changes over the next few releases, using new processes.

Sincerely,

Morten KrombergCTO, Dyalog Ltd.

APL Seeds ’21: Wednesday 31 March

Last Wednesday we hosted APL Seeds ’21, an event for those just starting their APL journey. Although we knew we had a good programme with some exceptional presenters in place, we very quickly had to increase our Zoom webinar limit to accommodate the 287 people who registered to attend! We were surprised and excited by the demographic, spanning 32 countries and with the vast majority having only basic or no APL experience.






The meeting started with a brief introduction from Dyalog’s Managing Director, Gitte Christensen, who shared her initial “Eureka” APL moment and gave some examples of situations in which APL is used today. Richard Park then took us on a whirlwind tour of APL’s past (including the very cool 1975 APL demonstration!) before demystifying the “beautiful squiggles” that define APL and introducing the modern resources that are available for learning APL (for a summary of these see the suggestions for learning resources available on our website).

The main presentations began with Rodrigo Girão Serrão giving a basic introduction to APL functions and syntax. Using the example of manually justifying text, he showed just how natural it is to process data in arrays by combining a few functions and operators. His initial exploration using a small snippet of text worked instantaneously and without issue on a whole book. After seeing this, hopefully you’ll feel an urge to learn some more – either because you got hooked (like Rodrigo did!) or simply because you want to learn how to think in an array-oriented way, which is very relevant in many situations today, such as when working with GPUs.

Martin Janiczek used a real-life example from the market insights and consumer trends company that he works for (GWI). Despite being a self-described “APL baby”, having learned APL for only around a month, he was able to get to grips with the tree structures that he wanted to use, and talked about how learning APL led him to change his overall approach to the problem. He achieved a highly-performant working prototype in two weeks and with only 172 lines of code, despite starting from the position of a complete APL beginner. Although ultimately his APL model was not taken into production, it inspired a complete rethink and new approach in the eventual product.

Conor Hoekstra (NVIDIA) describes himself as “not an APLer but a big fan”, and his enthusiasm is obvious and contagious! His explorations in APL are YouTube famous, and here again he deftly shows how APL can be used as a tool of thought to explore problems from many different angles with relative ease. He went through multiple different solutions to writing an All Equals function (is every element in a list the same as every other element?), playing with different primitives and comparing the performance of the solutions.

The final presentation came from Tomas Gustafsson, creator of the stunning Stormwind boating simulator. Tomas introduced the technology behind the 3D engine that he uses for his simulator, explaining the code that makes it all happen, before walking us through creating some simple 3-D shapes (a rotating triangle and an icosahedron) and the pitfalls that this entails. From the comments we know he seemed to inspire several members of the audience to want to know more… so for those of you that do, his code examples will soon be available from the APL Seeds web page.

We hope everyone found the event useful and enjoyable (the feedback seems to indicate that you did – thank you!). Relevant materials have started to be uploaded to the APL Seeds ’21 webpage – this page also includes links to recordings of the presentations, which are all on dyalog.tv:

 

Welcome Rodrigo Girão Serrão

The story of how Rodrigo got his first internship at Dyalog is, in his opinion, a textbook example of serendipity. As 2020 started, Rodrigo began actively participating in an online code golf community, where people try to solve programming challenges in as few bytes of code as possible. Whilst his golfing skills were possibly lacking, the challenges he posted were usually well accepted. Posting many challenges meant Rodrigo got exposed to answers in all sorts of programming languages, from C, Java, Python and JavaScript, to Jelly, 05AB1E, Husk…and APL. Because of the context and the aspect of it, Rodrigo first thought APL was one of those “esolangs” and not a serious programming language.

Rodrigo’s fascination with APL led him to start frequenting The APL Orchard chatroom, where a small number of brilliant people convened to discuss all things APL. Here he met Adám Brudzewsky, who was keen on teaching APL to newcomers, and so began Rodrigo’s journey to learn APL.

His interest in APL kept growing, and he found it to be a simple and expressive language that also incorporated his affinity with mathematics. One day, while lurking in The APL Orchard, Adám asked Rodrigo if he would be interested in taking an intern position at Dyalog…a few emails later it was established that Rodrigo would work as a part-time intern at Dyalog during the Summer of 2020. This enabled Dyalog to make the most of Rodrigo’s skills in teaching and technical writing, and meant Rodrigo could indulge his passion for sharing knowledge about mathematics and programming while still finishing his MSc in Applied Mathematics. After his internship, Rodrigo took some time to complete his MSc thesis before returning to Dyalog to finish what he had started and hopefully to take part in many other interesting projects. When he is not working for Dyalog, Rodrigo may be found leading a Portuguese APL meetup, writing a blog post for his website (mathspp.com), or maybe leading a workshop or course. Other than working, Rodrigo likes to spend time with his loved ones, read fantasy books, eat chocolate, and watch silly comedy movies.

Welcome Shuhao Yang

Image

Shuhao joined Dyalog straight after he completed his Master’s degree in quantum computing, which happened to be during the second COVID-related lockdown in the UK. This has given him a quite unusual experience of starting a career as he has yet to meet a single member of the Dyalog team face to face! Shuhao obtained his Bachelor’s degree in mathematics – although he was interested in computer science, he studied mathematics as a way of looking for the root of CS and computing. He has broad interests across different software including Matlab, Python and LaTeX and has developed a solid knowledge on C++.

Shuhao enjoys the romantic theories in computer science and always wanted to work in one of the summit areas of CS – compilers, graphics and operating systems. He’s very happy that he now has the opportunity to work on the Dyalog APL interpreter.

Highlights of the 2020 APL Problem Solving Competition – Phase I

We received some excellent competition entries this year. Once again, thank you to all those who participated, congratulations to this year’s winners, and thank you to the Grand Prize Winner Andrii Makukha for his presentation at this year’s user meeting.

This post contains some suggested Phase I solutions along with some comments from the judges. Before each solution there is a brief summary of the problem and a link to the full problem on the practice problems site; you can also download the full set of problem descriptions as a PDF.

This page contains spoilers. The suggested solutions are not the only correct solutions, and may not necessarily be best practice depending on your application or most optimal in terms of performance, but they do solve the problems in some particularly interesting and elegant ways. If you’d like to try to solve the problems yourself before looking at these example solutions, you can use problems.tryapl.org to check your solutions.

1: Let’s Split

The first problem was to write a function splitting a vector right argument into a nested vector of vectors according to a signed integer left argument. For example:

      ¯3 (your_function) 'DyalogAPL'
┌──────┬───┐
│Dyalog│APL│
└──────┴───┘

Most entrants successfully solved this problem using dyadic take and drop .

{c←⍺+(⍺<0)×≢⍵ ⋄ (c↑⍵)(c↓⍵)}

It was common to use the left argument as-is with and , and swap the two parts of the result using ⌽⍣condition or condition⌽array, but the solution above avoids the swap by computing appropriate arguments to and .

Try it now with TryAPL

2: Character Building

In problem 2, we asked participants to partition a vector of UTF-8 character encodings, similarly to 'UTF-8'∘⎕UCS¨, without using ⎕UCS. For example:

      (your_function) 68 194 165 226 141 186 226 140 138 240 159 148 178 57   
┌──┬───────┬───────────┬───────────┬───────────────┬──┐
│68│194 165│226 141 186│226 140 138│240 159 148 178│57│
└──┴───────┴───────────┴───────────┴───────────────┴──┘

Eight participants submitted this (or a variation thereof):

{⍵⊂⍨1≠128 192⍸⍵}

Instead of doing multiple comparisons, this neatly uses interval index to check which range the argument is in. It then uses partitioned-enclose to create partitions beginning where code points are either below 128 or above 192.

Try it now with TryAPL

3: Excel-lent Columns

Problem 3 was simply to convert Microsoft Excel-style column letters to an integer. For example:

      (your_function) 'APL'
1104

Thirty-five participants submitted variations on this:

{26⊥⎕A⍳⍵}

While simple at first glance, it is actually quite involved because ⎕A⍳⍵ can give 26 (for Z) which isn’t a valid digit in base-26. However, decode handles out-of-bounds digits by carrying.

Try it now with TryAPL

4: Take a Leap

The task for problem 4 was to write a function to verify which of an array of integer years are leap years. For example:

      (your_function) 1900+10 10⍴⍳100
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1
0 0 0 1 0 0 0 1 0 0
0 1 0 0 0 1 0 0 0 1

We had eight solutions like this one:

{0≠.=4 100 400∘.|⍵}

At first, it generates a 3-element vector showing whether the argument is divisible by 4, 100 or 400.

      0=4 100 400∘.|1900
1 1 0

The cleverness then is that ≠⌿ is used to express the logic of the leap-year algorithm. From Wikipedia:

if (year is not divisible by 4) then (it is a common year)
else if (year is not divisible by 100) then (it is a leap year)
else if (year is not divisible by 400) then (it is a common year)
else (it is a leap year)

We can check this with all possible length-3 boolean arguments:

      2⊥⍣¯1⍳¯1+2*3
0 0 0 1 1 1 1
0 1 1 0 0 1 1
1 0 1 0 1 0 1
      ≠⌿2⊥⍣¯1⍳¯1+2*3
1 1 0 1 0 0 1

Consider each case in turn:
1. Leap year, return 1
2. Can never occur
3. Not a leap year, return 0
4. Can never occur
5. Can never occur
6. Can never occur
7. Leap year, return 1

It is good because it uses no explicit loops and keeps intermediate values flat (no nesting). The solution leverages that each leap year rule is an exception to the previous one, and this particular formulation employs an unusual inner product ≠.= (equivalent to {≠/⍺=⍵} for vector arguments) to compute the parity of the divisibilities.

Try it now with TryAPL

5: Stepping in the Proper Direction

Problem 5 was to create a list generator somewhat similar to iota . However, this list generator takes a 2-element integer right argument and returns a list starting from the first integer and either increasing or decreasing in steps of 1 until the last integer inclusively. For example:

      (your_function) 4 ¯3
4 3 2 1 0 ¯1 ¯2 ¯3

Only one person had this exact solution, though many solutions were not too far off:

{(⊃⍵)-0,(××⍳∘|)-/⍵}

This dfn contains a 3-train or fork. Having seen contestants use the following format before, we feel compelled to provide you with a commented version of the above:

{
               -/⍵   ⍝ The length of the result is 1 more than the difference
           ⍳∘|)      ⍝ Integers up to the absolute difference
          ×          ⍝ times
        (×           ⍝ The sign of the difference
      0,             ⍝ Make the range inclusive
     -               ⍝ Use arithmetic to compute the correct result
 (⊃⍵)                ⍝ From the first value
                  }

Alternatively:

{
 (⊃⍵)                ⍝ From the first value
     -               ⍝ to
      0,             ⍝ inclusively
        (×           ⍝ The sign of...
          ×          ⍝ times
           ⍳∘|)      ⍝ Integers in the range of...
               -/⍵   ⍝ The difference
                  }    

This one excels in only computing necessary values once, and cleverly adjusts the generated values to rise or fall as needed, using the sign of the difference between the beginning and end points of the target range.

Try it now with TryAPL

6: Move to the Front

The task for problem 6 was to move all elements in the right argument vector equal to the left argument scalar to the start of that vector. For example:

      'a' (your_function) 'dyalog apl for all'
aaadylog pl for ll

Only one participant found this train, though two others submitted dfns using the same idea:

∩⍨,~⍨

Instead of computing indices or selecting elements, this simply employs two set functions, intersection and without ~. The asymmetry of intersection, namely that it preserves duplicates from its left argument, is here used to great advantage.

Try it now with TryAPL

7: See You in a Bit

Problem 7 involved writing a function to compare set bits in the base-2 representations of its integer arguments. For example:

      2 (your_function) 7   ⍝ is 2 in 7 (1+2+4)?
1

Eleven solutions used this method:

∧/(≤/2⊥⍣¯1,)

Indeed, the problem is about finding particular set bits in a binary number, hence the 2⊥⍣¯1. The overall function is a 2-train or atop, where the right-hand function is itself a 3-train or fork.

We can break it down as follows:

∧/             ⍝ Are all of (0 if any are *not*)
  (≤/          ⍝ Set left bits also set in right
     2⊥⍣¯1     ⍝ in The base-2 representation of
	      ,)   ⍝ The left and right arguments?

The function less than or equal to only returns 0 where a left bit is not found in the right argument:

      5((⊢,⍥⊂⍪⍤(≤/))2⊥⍣¯1,)9   ⍝ Just a fancy way of visualising the intermediate and final result
┌───┬─┐
│0 1│1│
│1 0│0│
│0 0│1│
│1 1│1│
└───┴─┘

This is pretty impressive, as it both demonstrates array orientation (in treating both arguments together) and uses Dyalog APL’s fancy inverse operator ⍣¯1 to use as many bits as necessary, while keeping the two representations aligned.

Try it now with TryAPL

8: Zigzag Numbers

The solution to problem 8 returns a 1 if its integer argument’s digits consecutively rise and fall throughout. For example, 12121 is a zigzag number, but 1221 is not.

We saw a handful of solutions of this type:

∧/0>2×/2-/10∘⊥⍣¯1

We can decompose it like so:

∧/                  ⍝ Are all
  0>                ⍝ Negative for...
    2×/             ⍝ Consecutively different signs of...
       2-/          ⍝ The pairwise difference of...
          10∘⊥⍣¯1   ⍝ The digits of the input?

It constitutes a good example of how the pattern in trains often is a natural one (this is actually an 8-train), and also shows off two uses of pairwise application 2f/ to compute the pairwise difference (the sign of which indicates the direction from digit to digit) and then the pairwise product (which due to the rules for multiplication of signed numbers indicates if a change has happened or not).

Try it now with TryAPL

9: Rise and Fall

Problem 9 involved writing a function to verify if a numeric vector has two properties:

  • The elements increase or stay the same until the “apex” (highest value) is reached
  • After the apex, any remaining values decrease or remain the same

For example:

      (your_solution)¨(1 2 2 3 1)(1 2 3 2 1)(1 3 2 3 1)
1 1 0

Actually, nobody had this exact solution, however, a handful came very close:

⊢≡⌈\⌊∘⌽⌈\∘⌽

Instead of trying to analyse the numbers, it does a running maximum from the left and from the right. If the minimum of those matches the original numbers, then we have exactly one peak.

⊢             ⍝ The input vector
 ≡            ⍝ matches
  ⌈\          ⍝ The max-scan from the left (msl)
    ⌊∘⌽       ⍝ The lower of msl and ⌽msr
       ⌈\∘⌽   ⍝ The max-scan from the right (msr)

We can visualise ⌊∘⌽ by stacking its arguments on top of one another:

      1 3 5,[0.5]⌽2 5 4
1 3 5
4 5 2
      ⌊⌿1 3 5,[0.5]⌽2 5 4
1 3 2
      1 3 5⌊∘⌽2 5 4
1 3 2

When used with the two max-scans, we can see how this solution works.

      (⌈\,[0.5]∘⌽⌈\∘⌽)1 3 2  
1 3 3  
3 3 2  
      (⌈\,[0.5]∘⌽⌈\∘⌽)1 0 2  
1 1 2  
2 2 2

Try it now with TryAPL

10: Stacking It Up

The task for problem 10 was to format a nested vector of simple arrays as if displayed using {⎕←⍵}¨, and then to return the formatted character matrix ({⎕←⍵}¨ simply returns its argument). For example:

      (your_function) (3 3⍴⍳9)(↑'Adam' 'Michael')(⍳10) '*'(5 5⍴⍳25)
1 2 3               
4 5 6               
7 8 9               
Adam                
Michael             
1 2 3 4 5 6 7 8 9 10
*                   
 1  2  3  4  5      
 6  7  8  9 10      
11 12 13 14 15      
16 17 18 19 20      
21 22 23 24 25

We had a couple of entries like this:

{¯1↓∊(⍕¨⍵),¨⎕UCS 13}

This was a tricky problem, especially as the automated testing didn’t include the 'a'1 test case, and many didn’t catch that one. Whilst most people wrote complicated code to get matrices for the element arrays, two participants thought outside the box, and simply joined the arrays with newlines after converting them to text.

Try it now with TryAPL

Conclusion

As always, not only are we deeply impressed by the ingenuity and cleverness of your submissions, but we also continue to be amazed by the number of people successfully solving most, if not all, of the problems in Phase I.

If you’d like to be notified when next year’s competition launches, go to dyalogaplcompetition.com and submit your email address.

Welcome Ron Murray

Ron flying in 2003

Ron Murray is a recent addition to the Dyalog team, with a long history in the APL community. He first encountered APL/360 in 1969 and was hooked. He used it as the basis for teaching Computer Science courses for the Hampton, Virginia High Schools. Then, working with other APL pioneers, he wrote several APL applications and contributed to five different APL implementations at The Computer Company, STSC, Burroughs, Data Resources, and Analogic Corporation.

From 1986 until 2019 he left the world of APL to develop software on Microcomputers for Microsoft and Amazon, where he contributed to various development projects for Windows, OS/2, NT, Visual Basic, Encarta, and a variety of projects within the Microsoft Research Division as well the Developer Relations Group. He also contributed to the scalability and reliability of the Amazon transaction accounting system and the Windows Azure Archival Storage System.

He also ran an Aviation business for several years at the Tacoma Narrows airport, and started an internet television company with three friends. Together they learned a lot about crawling the web using machine learning, categorizing videos by their subject matters and quality, as well as constructing interactive user interfaces on IOS devices.

During all that non-APL work he continued to use APL as a tool of thought for organizing, analyzing, and clarifying the work that needed to be done.

Since July of 2020 he’s been applying the many non-APL things he’s learned to help extend and improve the Dyalog APL systems and their interactions with the rest of the computing world.

He points out that Windows 95, which is now 25 years old is about half as old as the APL/360 release!