News about Dyalog

Jul 21, 2020

Introducing Dyalog Version 18.0 Issue 2


The first issue of Dyalog version 18.0 was made generally available on 10 June 2020. It was released with a few loose ends that we hadn't quite managed to tie down but felt were not worth holding the entire release back for. These have now been resolved. The most prominent changes in Issue 2 are:

  • Notarisation of the version for Apple macOS: Issue 2 of Dyalog version 18.0 is notarised and should be easy to install and use on macOS Catalina without requiring the manual work-arounds that Issue 1 needed. Note that when you first attempt to access certain folders, such as your Documents folder, macOS will prompt you to allow Dyalog to perform this action; this is intended behaviour from Apple.
  • The files required to support the Zero Footprint RIDE (where the interpreter serves the RIDE up as a web server without requiring a local installation of RIDE) are now included on all non-Windows platforms (see dyalog_readme.htm or the Ride User Guide for more information).
  • A new set of Docker images based on version 18.0 has been posted to dockerhub.com.

In addition, despite our best efforts to test new releases of APL before we release them, and extensive Beta testing that we are grateful to our users for helping us with, there is always a flurry of bugs that are only found when people start using a new release in production. Due to the extensive tuning work done for Dyalog version 18.0, some of these bugs were in APL primitives. Some of these had the potential to return incorrect results under circumstances that were so rare that they were not discovered during testing, specifically:

18261: Membership () could return incorrect results when both arguments were 4-byte integers and SSE instructions are not available

18264: on long Boolean vectors could return incorrect results

18329: Simultaneously dropping both rows and columns from certain sizes of Boolean matrices returned a result with an incorrect block of leading zeros in the first few elements

In addition to fixing issues reported in version 18.0, the development team has spent significant time since the release of Issue 1 working on some of the most important issues in the backlog (rather than waiting until the end of the next cycle before doing this kind of work). The results of this work are included in Issue 2.

Dyalog VERY STRONGLY RECOMMENDS that you install Issue 2 of version 18.0 as soon as possible. Although DSS contract holders can apply the latest patches on Microsoft Windows as usual, we would still recommend installing Issue 2 as the patches only affect the interpreter itself. You can download version 18.0 Issue 2 from https://www.dyalog.com/download-zone.htm or, if you have a valid DSS contract, from https://my.dyalog.com.

 

WARNING: Although Dyalog version 18.0 is recommended for most users, we have recently been made aware of an issue with version 18.0 that can lead to a change in behaviour if your application makes calls from a lexically-scoped environment (dfns) to dynamic scope (tradfns or using the execute primitive) – and back again. If you believe that your application might be doing this, then we recommend that you delay upgrading to version 18.0. If you have already upgraded, you might want to consider reverting to version 17.1, or discussing the matter with us to understand whether your code could trigger the issue. More details about this issue follow below.

Issues with mixed Lexical and Dynamic Scope

We have discovered a potentially serious issue in version 18.0 which can strike applications that use dfns when temporarily switching from a lexically-scoped environment to a dynamically-scoped environment and back. Examples of such temporary switches to dynamic scope include:

  • Using execute () within a dfn and causing another dfn to be invoked.
  • Using a traditional operator to invoke one dfn from another.

In Dyalog version 18.0, use of the above types of execution path can lead to spurious value errors on return to the calling dfn. There is also a possibility, if both dfns have local variables with the same name, that a variable in the calling environment can assume the value that it had in the invoked scope.

This issue introduces the risk that applications that worked under version 17.1 will fail under version 18.0, or, in the worst case, produce incorrect results.

If you suspect that you are using execute or tradops from within dfns as described above, we recommend that you DO NOT UPGRADE TO VERSION 18.0 until the problem is resolved. We are currently unable to estimate when a fix will be available.

Background

dfns (and dops) are a form of functions and operators that were introduced into Dyalog to facilitate functional programming, using lexical scope. The notation has other attractions like automatic localisation and in-line anonymous functions. These attractions have made dfns popular for use within applications that are fundamentally procedural or even object oriented. Strict lexical scope is often inconvenient, and some applications take advantage of the fact that you can defeat the barriers imposed by lexical scope by temporarily switching to a traditional function or operator. Additionally, the use of the execute primitive to invoke another dfn acts as a temporary switch to dynamic scope and back.

Version 18.0 includes a fix for a bug in the lexical scope implementation that was initially reported in 2004. The infamous "Mantis 17" issue, which for many years had the lowest number of any open issue, has the following description: An inner dynamic function called from an external dynamic operator can produce a VALUE ERROR. John Scholes, the inventor of dfns, struggled with this problem over several years, and with the help of his colleagues he finally implemented a fix in late 2018. The fix was not included in version 17.1, which was supposed to have minimal language changes, but is part of version 18.0. The fix seems to work fine for the lexical scope cases that it was designed to resolve. Unfortunately, after shipping version 18.0, we have discovered that the fix interferes with the "temporary dynamic scope trick" that some users have come to rely on.