11 April 2023 – A Day to Celebrate!

Today we reach two very significant milestones.

40 Years of Dyalog APL

On this day, we have cause for celebration: it is 40 years since the release of Dyalog version 1.0! Geoff Streeter would say that from his perspective we are already in the 42nd year, as he and John Scholes started work on the new interpreter in 1981. On the other hand, Pete Donnelly might argue that the interpreter wasn’t really ready for serious use until a few years after that date. The fact remains that the APL ’83 conference in Washington DC saw the first official release of the product, and is considered to be the “birth” of Dyalog APL.

Farewell Geoff Streeter

On the same day, we also congratulate Geoff Streeter, the last Dyalog developer who will have worked on the interpreter throughout its existence, on the first day of his retirement. We are happy to be able to report that, unlike John Scholes who designed and built version 1.0 together with Geoff but sadly passed away in 2019, Geoff is retiring in good health. We wish him many happy years in retirement in the company of his wife Sarah, children and grandchildren – although we are also hoping that he will pop by the Dyalog office from time to time to let us know how he thinks we are doing, and hopefully also join us at some future user meetings!

Continuity

It is impossible to exaggerate the value of the dedication and continuity that Geoff and John provided to Dyalog over these four decades. In the early days, when company finances were shaky, they sometimes continued work on the interpreter without compensation. Today, we are still blessed with many team members who have worked for Dyalog for most, if not all, of their careers – although we are doing what we can to avoid the need for the extremes of dedication that were required in the early days.

Good Choices

Dyalog APL started its life as a unique combination of what was to become the leading nested array paradigm (APL2/NARS floating arrays) coupled with what John and Geoff (and the rest of the consulting team at Dyadic Systems Ltd) thought were the best “commercial” extensions selected from APL systems developed by STSC and I.P.Sharp Associates (component files and control structures from STSC, error trapping from IPSA, and many other features).

The early choices have stood the test of time, and paved the way for Dyalog to become extremely competitive when Windows 3.1 and John Daintree arrived at the same time – resulting in the extremely easy-to-use Win32 GUI support. In typical fashion, the team did not merely implement a tool for GUI programming, they adopted an approach that led to the very general notion of namespaces, which meant that the same architecture could be used to interface to COM/OLE and subsequently .NET and complete support for object-oriented programming in Dyalog.

UNIX

Geoff was (and is) very much a UNIX man, and the first versions of Dyalog were built for the small UNIX machines that Dyadic Systems expected to take over from the mainframe. Unfortunately, UNIX was slow to gain acceptance and, around 1990 when UNIX was finally starting to take off, Microsoft Windows arrived and the centre of gravity of the commercial business shifted in that direction. Geoff was often heard muttering to himself about how the company was making poor technical choices, driven by what he referred to as “commercial” pressure. He kept his head down, and ensured the UNIX implementations were well supported and that all designs took the needs of these platforms into account.

Users of UNIX-based Dyalog can be grateful for Geoff’s unwavering enthusiasm. And Dyalog too: not only is the IBM AIX version of APL still a significant source of revenue, the ease with which we have been able to add support for Linux and macOS is very much down to our long UNIX history and the goal of maintaining cross-platform compatibility, throughout the history of the product.

Dedication

As a further example of Geoff’s dedication: as a young man with a motorcycle and a keen sense of community, Geoff started riding at night, delivering blood from blood banks to hospitals where it was urgently needed. He started doing this in 1980, just after he started at Dyadic Systems, and, although he is retiring from Dyalog today, Geoff will continue as a volunteer for SERV S&L. Although he no longer rides his bike, he now acts as the controller for the new generation of riders. In October 2021, SERV S&L was presented with The Queen’s Award for Voluntary Service by the Lord Lieutenant of Surrey.

The Future

In the same way that Geoff has not been riding his bike at night, he has not been doing any new development on Dyalog for the last year. Instead, he has been preparing and holding internal presentations to a new generation of Dyalog developers, providing insight into the work that he has done over the last four decades. Ultimately an impossible task of course, but at least they now know where to start digging – and Geoff is still around to answer questions in an emergency if we offer him a cup of coffee and three plain chocolate digestives (his stated minimum requirement to come into the office!).

As Dyalog enters its fifth decade, all parts of Dyalog Ltd. (including the development team) are larger – and broader – than they have ever been. The good choices made by Geoff, John, Pete, and many others in the early years are holding up, and the company continues to grow.

Many thanks to Geoff from everyone at Dyalog Ltd. We will do our best to allow him to relax, enjoy his well-deserved retirement, and look forward to continued dividend payments from the Dyalog shares in his pension fund!

References:

Not All Heroes Wear Capes

On Friday 15 October, the SERV S&L was presented with The Queen’s Award for Voluntary Service by the Lord Lieutenant of Surrey. Geoff Streeter was one of 20 volunteer SERV S&L members who attended.

What Is SERV S&L?

SERV S&L (Service by Emergency Rider Volunteers for Surrey and London) are a charity organisation, made up entirely of volunteers, comprising motorbike riders, car drivers, controllers, and fundraisers. They transport blood products, urgent samples, medical supplies, and donated breast milk to hospitals and milk banks across Surrey & London, as well as carrying out a daily delivery of blood to the Air Ambulance service that covers Kent, Surrey, and Sussex. They support the regular delivery rounds that the NHSBT (National Health Service Blood and Transport) have in place; unlike the NHSBT, SERV S&L also operate throughout the night. All of this is provided free of charge to the NHS, releasing more money for patient care.

What Is The Queen’s Award for Voluntary Service?

QAVS (The Queen’s Award for Voluntary Service) celebrates the outstanding work of local volunteer groups across the UK. Created in 2002 for Queen Elizabeth II’s Golden Jubilee, QAVS awards shine a light on the fantastic work of voluntary groups. QAVS awards are the highest awards given to local voluntary groups in the U.K. (they are the equivalent of a personal MBE) and they are awarded for life.

Geoff’s Involvement with SERV S&L

A Personal Recollection

At the end of 1980, Paul McCann had a relation who could not get an urgent sample transported to the testing lab until the next morning. He was frustrated by this and organised a meeting to see what could be done, the result of which was that a group of advanced motor cycle trainers from a (now defunct) group called Star Rider decided to try to run a delivery service for blood/samples at night. I was not at that meeting but I heard about it from a fellow member of the Laverda Owners group; I made it to the second meeting (on 8 December 1980) and have been involved ever since. We obtained a room with a couple of bunks in a wooden building owned by MEFAS (Malden Emergency First Aid Society) and a telephone line, and started operating in early 1981.

The main distribution point for blood is located in Tooting and serves London, Kent, Surrey and Sussex (we have partner organisations in Kent, Sussex and Wessex). We do a main nightly run with typically 6 to 10 boxes down to an arranged change point for Kent and Sussex. We also partner with similar organisations across the U.K., and have occasional relay runs, for example, from Edinburgh to central London (I think that’s the longest that we’ve been involved in). More common are runs from Bristol. We typically shift 20 boxes a night and samples in the other direction and have about 8 riders/drivers on shift every night.

Financially, we get support from some Masonic Lodges and business groups. They prefer to buy bikes for us, and Citroen have given us a car (DS3) on permanent loan. We are in the process of acquiring/refurbishing a scout facility in Sutton to provide a base for the bikes/cars/van as well as for volunteers who live on the periphery of the area. We also raise funds by box waving outside supermarkets, garden centres, Brooklands, Waterloo Station, etc.

I started with the group as a biker, and used my Laverda 750, Laverda 1200 and Honda 650 Turbo to deliver blood and samples from 1981 until 1990, when I switched to car deliveries (which I continued to do until last year). I also acted as Treasurer from 2006 until 2010. I have been one of the controllers right from the start [Ed.: Controllers orchestrate the logistics of a shift; hospitals and partner groups place their orders and riders and drivers are dispatched as required – accurate scheduling and data logging are required to ensure efficient co-ordination and communication so that each run can be completed reliably], a role that has changed a lot over the last 40 years. In the early days controllers needed to be physically present with the one rider and the telephone. Then we moved to using pagers (but still needed to be present in the hut/sports centre) before everything changed with the advent of mobile phones – I now control from home. The expectation is that volunteers do one night a fortnight, but a shortage of volunteers relative to growing demand means that for a few years now I have been doing at least one shift a week.

Final Word

Congratulations Geoff, 40 years of volunteering for such a worthy cause is a fantastic achievement. All of us at Dyalog Ltd are really proud of your contribution.

To find out more about the amazing service provided by SERV S&L, including how to make a donation, visit https://servsl.org.uk/.

Announcing the Beta Programme for Dyalog APL Version 18.2


I am very pleased to be able to announce the start of the Beta testing of the next release of Dyalog APL! As explained in June, we decided to delay the release of version 18.1 in order to take a closer look at some of the optimisations that had been implemented in 18.0 (and were therefore also present in 18.1).

Our analysis of the optimised code concluded that, due to the nature of many of the new algorithms involving modern vector instructions and code generated from templates, we need significantly more time to create tests that will cover absolutely all the different cases that have been implemented. As a result, we do not feel that it is in the best interests of our users to release v18.1 in its current form.

The good news is that the features of modern source code management systems, combined with our collection of regression tests, have allowed us to create a new version of Dyalog APL that contains all of the new features added to versions 18.0 and 18.1. This includes bug fixes made during these two development cycles, but not the optimisations that make us feel uncomfortable. To differentiate this version from the existing version 18.1, we have decided to call the new version 18.2.

Everyone who was signed up for the v18.1 beta programme should now be able to download v18.2 beta. If you are not signed up as a beta tester already but would like to help us with testing, please get in touch. Under Microsoft Windows, testing should be significantly simpler this year, as we have started producing Microsoft Patch files (MSP) as the delivery mechanism for updates – something I have personally been looking forward to since before I joined Dyalog Ltd!

Version 18.2 Performance

One unavoidable consequence of the above is that the performance of v18.2 is closer to that of v17.1 than to v18.0. Our own tests show that we have not been pushed ALL the way back to “square one”: v18.2 appears to be slightly faster than version 17.1. Once v18.2 has been delivered we will work on carefully re-implementing the most valuable optimisations that have been removed. We welcome your input on which primitives you think we should speed up first, so please participate in testing v18.2 and let us know what you think we should prioritise as we start work on version 19.0!

Recommendations regarding Version 18.0

Over the summer, only one additional defect related to performance optimisation was discovered and fixed. We are not currently aware of any outstanding defects in v18.0 caused by recent optimisations. Dyalog Ltd is committed to providing support for version 18.0 until the arrival of the 3rd release following it, in accordance with normal policy.

However, if you have not yet upgraded to v18.0, Dyalog Ltd strongly recommends remaining on your current version and moving directly to v18.2 when it is released. If all goes well, this will happen at the end of 2021 or very early in 2022. If you are already using v18.0, then we recommend that you make plans to start evaluating v18.2 and moving to it as soon as possible.

Conclusion – and Apology

We are painfully aware that the defects found in v18.0 and the resulting uncertainty have seriously inconvenienced some of our users, and I apologise for this. The root cause is the growth and rejuvenation of the Dyalog development team. Our original processes for quality assurance relied on years of tacit knowledge; when enthusiastic new team members break significant new ground, more explicit planning and QA processes are required to make sure that new approaches are safe and stable.

When we resume work on optimisations following the release of v18.2, this will be done according to new guidelines that require the process to begin with a careful risk/benefit analysis of any enhancement to primitive functions. We will do everything that we can to move forward in a way that will allow us all to eventually look back on the events of 2021 as a significant step towards a more capable and reliable development organisation and product.

After all, in another two years it will be time to celebrate 40 years of Dyalog APL!

Thank You Ian Sharp

On July 16th, one of the most influential founders of what we today refer to as the “array language community” died peacefully, a few months after being diagnosed with lung cancer (link: Toronto Globe and Mail).

Ian Patrick Sharp

In 1964, Ian Patrick Sharp formed I.P.Sharp Associates (IPSA), together with six colleagues who were made redundant when Ferranti-Packard closed its computer division in Toronto, Canada. As Ian explains in a wonderful interview that was recorded in 1984 (link: Snake Island website), he was approached by people who wanted to recruit the whole team. Instead, he decided to form a company, since the team obviously had significant value.

The company was involved in the first APL implementation at IBM (APL\360). Subsequently, IBM allowed them to modify and enhance the system, and built a timesharing service that became known as SHARP APL. Roger Moore was a co-founder of IPSA and, in addition to being responsible for the supervisor that made SHARP APL a superior timesharing system, Roger was the chief architect of IPSANET, one of the worlds first packet switched networks.

In the late 1970s the combination of APL and IPSANET was revolutionary, and IPSA quickly attracted business from global corporate clients who used SHARP APL for e-mail, reporting and analytics, and a rapidly-growing collection of financial timeseries data – all completely new technologies at the time. In particular, the transmission of data over telephone lines changed the world. Ian had many absurd encounters with telecom monopolies who tried to protect old business models or profit from the new technology (link: archive.org).

A Stylised Map of the I.P.Sharp Associates APL Time-Sharing Network

Ian’s management style perfectly matched – and drove – the revolutionary technologies. As Ian explains so eloquently and humorously in the interview, IPSA recruited talented people without necessarily having specific tasks in mind. Ian set the tone and direction and then let people get on with it, moving around in the background to get a sense of how things were going and making adjustments without ever making a fuss. IPSA was a fantastic place to work and attracted a wonderfully diverse (in the most modern sense of the word) collection of smart people who developed revolutionary tools, helped a lot of customers, had a lot of fun, and made money.

Ultimately, IPSA was creative, problem-oriented and customer-driven to the extent that it failed to respond to fundamental changes in the market in time. At the end of the 1980s the timesharing revenues suddenly faded, and the company was acquired by Reuters for its timeseries databases and more or less disappeared overnight. However, IPSA had acted as a fantastic breeding ground for technology and talent for a quarter century, and there are hundreds of people who fondly and gratefully remember Ian for the way that he allowed them all to grow.

I don’t think it is a coincidence that so many of the active array language organisations have key players who were once IPSA employees (some of them appearing in more than one place thanks to relationships forged a very long time ago 😊).

  • Jsoftware: Roger Hui, Eric Iverson, Chris Burke, Ken Iverson
  • Kx: Arthur Whitney, Simon Garland, Stephen Taylor, Chris Burke
  • Dyalog: Gitte Christensen, Morten Kromberg, Roger Hui, Brian Becker, Dan Baronet
  • Snake Island Research: Robert Bernecky

As always, Roger has collected anecdotes about IPSA, Ian and other people who worked there, which you can find on jsoftware.com/papers/SharpQA.htm.

My Own IPSA Story

In 1978, my dad was moving out of an apartment in Oslo. At the same time, XEROX insisted that IPSA open an office in Oslo to support their international business, and several Canadians arrived there. I helped move some furniture and, sensing a keen interest and real excitement in programming, the IPSA Oslo team offered me a free account to play with APL timesharing, if I was interested. I effectively became a piece of furniture in the IPSA office after school, and had keys to the office so I could come and go as I pleased. After a year or so, they started throwing me bits of real work to do and paying me for my time. I think I was 17 at the time.

In addition to working as an APL consultant and tool builder, one of the things I did in my spare time was to write a tool for myself that would compare the entire contents of the e-mail directory with its state at the end of the previous week. Since IPSA was 100% managed by e-mail groups, this allowed me to know instantly when a new office was opened, a significant new project was started, and, of course, when new employees joined the company. By using this technique of harvesting e-mail addresses and sending unsolicited e-mail when an interesting project or person joined, I found my future partner both at home and the office – Gitte, the current CEO of Dyalog Ltd – only about 500km away in the IPSA Copenhagen branch.

I spent about a decade at IPSA and, after its sudden disappearance, Gitte and I have been trying to recreate the IPSA atmosphere in every team that we have been a member of. In a very real sense, I owe not only my career but almost everything of value about my life to Ian Sharp and the warm and welcoming company that he created.

Thank You, Ian!

News and Recommendations Regarding Dyalog Versions 17.1, 18.0 and 18.1

Background

Dyalog version 18.0 contained the largest number of optimised algorithms in the history of Dyalog APL. Unfortunately, since its release, we have found that our process for review and testing of optimisations was insufficient to cope with the quantity and nature of the optimisations. Many of the new algorithms had edge cases that escaped both the existing regression tests and the new tests that were written as part of development.

In August 2020, after the issues were reported following the release of Dyalog version 18.0, we issued a caution against the use of version 18.0 in production. We added significant additional testing during September and October, and subsequently lifted the caution in November. In April 2021, a user encountered an additional defect leading to incorrect results in a rare case of membership (), and in June a crash in an extremely rare case of “where” (monadic ). Since then, our own internal testing has found a defect in interval index (dyadic ).

Current Status

We have made patches available as defects have been detected and fixed, and “Issue 4” of version 18.0, made available to all users on 22 June 2021, includes fixes for all known issues with optimised code.

Download Dyalog version 18.0 Issue 4: Commercial users  Non-commercial users

At this time, we are executing two high priority projects, one aimed at increasing safety in the short term, the other at completely eliminating problematic optimisations within a few months:

  • We have further increased our internal testing of version 18.0, to verify the correctness of the optimisations that are still in the distributed product.
  • We are holding back the release of version 18.1, which was originally scheduled for release this month, while we remove most of the optimisations that went into version 18.0. We hope to release version 18.1 before the end of the 3rd Quarter of 2021, but will not let it out the door until we have complete confidence in this new version.

Recommendations

Dyalog Ltd recommends that users of Dyalog APL take the following action:

  • If you are using version 17.1 or earlier, plan to skip version 18.0 and upgrade directly to version 18.1 when it becomes available.
  • If you are using version 18.0, apply the latest patches or reinstall using Issue 4 when it becomes available. Continue to monitor our DSS e-mail broadcasts and apply patches quickly if we should find and fix any further issues that you think might impact your application. When version 18.1 becomes available, plan to upgrade to it at your earliest convenience.

Support for Versions 17.1 and 18.0

Considering this extraordinary situation, we have decided to extend support for version 17.1 for one additional release cycle. In other words, version 17.1 will be supported until we release the 4th subsequent version of Dyalog APL. Note that we are about to provide an updated version 17.1 installer (“Issue 3”) which collects all updates to version 17.1 since the original release.

Version 18.0 will be supported for the normal number of cycles, that is, until we release the 3rd following version. However, Dyalog Ltd recommends upgrading to version 18.1 as soon as it becomes available and you are able to schedule the upgrade.

Non-Commercial Versions

Given the nature of these defects, it is our intention to update our non-commercial distributions of version 18.0, although this process may lag behind the distribution of patches to clients paying for support by some weeks.

Conclusion

We apologise for the significant inconvenience that we know this is causing for some of you and thank you for your patience. We have made significant changes to our internal procedures regarding risk assessment, verification, and testing requirements for changes to existing primitive functions, to avoid a recurrence.

We do expect to re-apply those optimisations that we believe provide significant performance benefits to justify the risk of making changes over the next few releases, using new processes.

Sincerely,

Morten Kromberg
CTO, Dyalog Ltd.

A Blustery Spring

Dyalog version 17.1 will be released soon, with the HTMLRenderer working under Windows, macOS and GNU/Linux, the “Link” system providing infrastructure for connecting APL to source code management systems, pre-built Docker containers with Dyalog APL for Linux installed and many other enhancements that simplify the installation and maintenance of systems based on Dyalog APL.

We’ll be writing much more about version 17.1 soon, and next year’s 18.0 release in due course. The main purpose of this blog entry is to let you know about new members of the Dyalog team and, unfortunately, a couple of departures as well.

Departures


In February, John Scholes passed away. Together with Geoff Streeter, John was one of the original implementors of Dyalog APL in 1982-1983, a cornerstone of all aspects of the Dyalog language and business, and one of the pillars of the APL community. Many members of the community have paid tribute to our Genius, Gentleman and Mischievous Schoolboy at http://johnscholes.rip.

At the end of May 2019, Jay Foad is leaving Dyalog to return to his first love (as a software developer) and become a proper compiler geek again, after nearly a decade of helping move Dyalog APL forward and, for the last three years, helping to “herd the cats” as CTO. We will sorely miss Jay’s technical excellence but understand the desire to hit the sweet skill spot when the opportunity arises, and we wish him good fortune in that pursuit! You can read Jay’s farewell blog post here.

Jay’s management responsibilities will be shared between Richard Smith, our Development Manager and myself; I will be re-assuming the role of CTO until further notice.

New Faces in 2019

The good news is that we will welcome several new people to Dyalog in 2019 – new hands to write code in APL, to work on the APL interpreter, and to write documentation and training materials to help new and old users get their work done more effectively.

APL Consultants

In response to client requests and to help new clients get started writing their first APL systems, we are creating a consulting group in the USA. To date, we have recruited two members for this team: Nathan Rogers joined the team at the end of April and is based in Denver, Colorado, and Josh David starts work for Dyalog in early June (as soon as he graduates) and will be based in New Jersey. If you think you have heard of Josh before, that is probably because he was a winner of the Dyalog Problem-Solving Contest in 2016 (https://www.dyalog.com/news/112/420/2016-APL-Programming-Contest-Winners.htm) – and a runner up in 2015. Nathan found us thanks to Adam Brudzewsky’s work on Stack Exchange: https://chat.stackexchange.com/rooms/52405/the-apl-orchard. You can reach them both using e-addresses in the form firstname at dyalog.com.

When members of the consulting team are not working for clients, the intention is that they will be members of the APL Tools Group at Dyalog, working on new tools for APL application development and helping create test suites for Dyalog APL. They will also support Richard Park, who joined us late in 2018, to work on the creation of training materials and tutorials for new users.

Once we have a better idea of the demand for consulting in North America, we expect to grow the team. Please let us know if you could use hired APL hands – in any territory! If we don’t have the resources ourselves, we may be able to find someone else.

Programming Language Implementors

Nathan comes to us with experience from a broad set of tools and programming languages. In addition to writing tools in APL, he will be a part-time member of the core development team, working on the APL interpreter and its interfaces in C, C#, JavaScript, Python and other languages. However, he won’t spend enough time on this to make up for the loss of Jay, who (like most managers at Dyalog) spent a significant amount of his time writing code.

Therefore, as described at https://www.dyalog.com/careers.htm, we are recruiting at least one C / C++ programmer to help us grow the core team.

A Busy – and Exciting Time

2019 is looking like an extremely busy year, with significant growth at Dyalog. As usual, our plan is to bring all the new (and old) hands to the Dyalog user meeting, which will be held in Elsinore, Denmark this year – September 8th to 12th. Details of the programme will soon start to appear at https://www.dyalog.com/user-meetings/dyalog19.htm. If you would like to present an APL-related experience to the user community, make proposals for new features of Dyalog products or suggest topics that you would like Dyalog to speak about at the user meeting, then please let us know as soon as possible!