Thank You Ian Sharp

On July 16th, one of the most influential founders of what we today refer to as the “array language community” died peacefully, a few months after being diagnosed with lung cancer (link: Toronto Globe and Mail).

Ian Patrick Sharp

In 1964, Ian Patrick Sharp formed I.P.Sharp Associates (IPSA), together with six colleagues who were made redundant when Ferranti-Packard closed its computer division in Toronto, Canada. As Ian explains in a wonderful interview that was recorded in 1984 (link: Snake Island website), he was approached by people who wanted to recruit the whole team. Instead, he decided to form a company, since the team obviously had significant value.

The company was involved in the first APL implementation at IBM (APL\360). Subsequently, IBM allowed them to modify and enhance the system, and built a timesharing service that became known as SHARP APL. Roger Moore was a co-founder of IPSA and, in addition to being responsible for the supervisor that made SHARP APL a superior timesharing system, Roger was the chief architect of IPSANET, one of the worlds first packet switched networks.

In the late 1970s the combination of APL and IPSANET was revolutionary, and IPSA quickly attracted business from global corporate clients who used SHARP APL for e-mail, reporting and analytics, and a rapidly-growing collection of financial timeseries data – all completely new technologies at the time. In particular, the transmission of data over telephone lines changed the world. Ian had many absurd encounters with telecom monopolies who tried to protect old business models or profit from the new technology (link: archive.org).

A Stylised Map of the I.P.Sharp Associates APL Time-Sharing Network

Ian’s management style perfectly matched – and drove – the revolutionary technologies. As Ian explains so eloquently and humorously in the interview, IPSA recruited talented people without necessarily having specific tasks in mind. Ian set the tone and direction and then let people get on with it, moving around in the background to get a sense of how things were going and making adjustments without ever making a fuss. IPSA was a fantastic place to work and attracted a wonderfully diverse (in the most modern sense of the word) collection of smart people who developed revolutionary tools, helped a lot of customers, had a lot of fun, and made money.

Ultimately, IPSA was creative, problem-oriented and customer-driven to the extent that it failed to respond to fundamental changes in the market in time. At the end of the 1980s the timesharing revenues suddenly faded, and the company was acquired by Reuters for its timeseries databases and more or less disappeared overnight. However, IPSA had acted as a fantastic breeding ground for technology and talent for a quarter century, and there are hundreds of people who fondly and gratefully remember Ian for the way that he allowed them all to grow.

I don’t think it is a coincidence that so many of the active array language organisations have key players who were once IPSA employees (some of them appearing in more than one place thanks to relationships forged a very long time ago 😊).

  • Jsoftware: Roger Hui, Eric Iverson, Chris Burke, Ken Iverson
  • Kx: Arthur Whitney, Simon Garland, Stephen Taylor, Chris Burke
  • Dyalog: Gitte Christensen, Morten Kromberg, Roger Hui, Brian Becker, Dan Baronet
  • Snake Island Research: Robert Bernecky

As always, Roger has collected anecdotes about IPSA, Ian and other people who worked there, which you can find on jsoftware.com/papers/SharpQA.htm.

My Own IPSA Story

In 1978, my dad was moving out of an apartment in Oslo. At the same time, XEROX insisted that IPSA open an office in Oslo to support their international business, and several Canadians arrived there. I helped move some furniture and, sensing a keen interest and real excitement in programming, the IPSA Oslo team offered me a free account to play with APL timesharing, if I was interested. I effectively became a piece of furniture in the IPSA office after school, and had keys to the office so I could come and go as I pleased. After a year or so, they started throwing me bits of real work to do and paying me for my time. I think I was 17 at the time.

In addition to working as an APL consultant and tool builder, one of the things I did in my spare time was to write a tool for myself that would compare the entire contents of the e-mail directory with its state at the end of the previous week. Since IPSA was 100% managed by e-mail groups, this allowed me to know instantly when a new office was opened, a significant new project was started, and, of course, when new employees joined the company. By using this technique of harvesting e-mail addresses and sending unsolicited e-mail when an interesting project or person joined, I found my future partner both at home and the office – Gitte, the current CEO of Dyalog Ltd – only about 500km away in the IPSA Copenhagen branch.

I spent about a decade at IPSA and, after its sudden disappearance, Gitte and I have been trying to recreate the IPSA atmosphere in every team that we have been a member of. In a very real sense, I owe not only my career but almost everything of value about my life to Ian Sharp and the warm and welcoming company that he created.

Thank You, Ian!

News and Recommendations Regarding Dyalog Versions 17.1, 18.0 and 18.1

Background

Dyalog version 18.0 contained the largest number of optimised algorithms in the history of Dyalog APL. Unfortunately, since its release, we have found that our process for review and testing of optimisations was insufficient to cope with the quantity and nature of the optimisations. Many of the new algorithms had edge cases that escaped both the existing regression tests and the new tests that were written as part of development.

In August 2020, after the issues were reported following the release of Dyalog version 18.0, we issued a caution against the use of version 18.0 in production. We added significant additional testing during September and October, and subsequently lifted the caution in November. In April 2021, a user encountered an additional defect leading to incorrect results in a rare case of membership (), and in June a crash in an extremely rare case of “where” (monadic ). Since then, our own internal testing has found a defect in interval index (dyadic ).

Current Status

We have made patches available as defects have been detected and fixed, and “Issue 4” of version 18.0, made available to all users on 22 June 2021, includes fixes for all known issues with optimised code.

Download Dyalog version 18.0 Issue 4: Commercial users  Non-commercial users

At this time, we are executing two high priority projects, one aimed at increasing safety in the short term, the other at completely eliminating problematic optimisations within a few months:

  • We have further increased our internal testing of version 18.0, to verify the correctness of the optimisations that are still in the distributed product.
  • We are holding back the release of version 18.1, which was originally scheduled for release this month, while we remove most of the optimisations that went into version 18.0. We hope to release version 18.1 before the end of the 3rd Quarter of 2021, but will not let it out the door until we have complete confidence in this new version.

Recommendations

Dyalog Ltd recommends that users of Dyalog APL take the following action:

  • If you are using version 17.1 or earlier, plan to skip version 18.0 and upgrade directly to version 18.1 when it becomes available.
  • If you are using version 18.0, apply the latest patches or reinstall using Issue 4 when it becomes available. Continue to monitor our DSS e-mail broadcasts and apply patches quickly if we should find and fix any further issues that you think might impact your application. When version 18.1 becomes available, plan to upgrade to it at your earliest convenience.

Support for Versions 17.1 and 18.0

Considering this extraordinary situation, we have decided to extend support for version 17.1 for one additional release cycle. In other words, version 17.1 will be supported until we release the 4th subsequent version of Dyalog APL. Note that we are about to provide an updated version 17.1 installer (“Issue 3”) which collects all updates to version 17.1 since the original release.

Version 18.0 will be supported for the normal number of cycles, that is, until we release the 3rd following version. However, Dyalog Ltd recommends upgrading to version 18.1 as soon as it becomes available and you are able to schedule the upgrade.

Non-Commercial Versions

Given the nature of these defects, it is our intention to update our non-commercial distributions of version 18.0, although this process may lag behind the distribution of patches to clients paying for support by some weeks.

Conclusion

We apologise for the significant inconvenience that we know this is causing for some of you and thank you for your patience. We have made significant changes to our internal procedures regarding risk assessment, verification, and testing requirements for changes to existing primitive functions, to avoid a recurrence.

We do expect to re-apply those optimisations that we believe provide significant performance benefits to justify the risk of making changes over the next few releases, using new processes.

Sincerely,

Morten KrombergCTO, Dyalog Ltd.

A Blustery Spring

Dyalog version 17.1 will be released soon, with the HTMLRenderer working under Windows, macOS and GNU/Linux, the “Link” system providing infrastructure for connecting APL to source code management systems, pre-built Docker containers with Dyalog APL for Linux installed and many other enhancements that simplify the installation and maintenance of systems based on Dyalog APL.

We’ll be writing much more about version 17.1 soon, and next year’s 18.0 release in due course. The main purpose of this blog entry is to let you know about new members of the Dyalog team and, unfortunately, a couple of departures as well.

Departures


In February, John Scholes passed away. Together with Geoff Streeter, John was one of the original implementors of Dyalog APL in 1982-1983, a cornerstone of all aspects of the Dyalog language and business, and one of the pillars of the APL community. Many members of the community have paid tribute to our Genius, Gentleman and Mischievous Schoolboy at http://johnscholes.rip.

At the end of May 2019, Jay Foad is leaving Dyalog to return to his first love (as a software developer) and become a proper compiler geek again, after nearly a decade of helping move Dyalog APL forward and, for the last three years, helping to “herd the cats” as CTO. We will sorely miss Jay’s technical excellence but understand the desire to hit the sweet skill spot when the opportunity arises, and we wish him good fortune in that pursuit! You can read Jay’s farewell blog post here.

Jay’s management responsibilities will be shared between Richard Smith, our Development Manager and myself; I will be re-assuming the role of CTO until further notice.

New Faces in 2019

The good news is that we will welcome several new people to Dyalog in 2019 – new hands to write code in APL, to work on the APL interpreter, and to write documentation and training materials to help new and old users get their work done more effectively.

APL Consultants

In response to client requests and to help new clients get started writing their first APL systems, we are creating a consulting group in the USA. To date, we have recruited two members for this team: Nathan Rogers joined the team at the end of April and is based in Denver, Colorado, and Josh David starts work for Dyalog in early June (as soon as he graduates) and will be based in New Jersey. If you think you have heard of Josh before, that is probably because he was a winner of the Dyalog Problem-Solving Contest in 2016 (https://www.dyalog.com/news/112/420/2016-APL-Programming-Contest-Winners.htm) – and a runner up in 2015. Nathan found us thanks to Adam Brudzewsky’s work on Stack Exchange: https://chat.stackexchange.com/rooms/52405/the-apl-orchard. You can reach them both using e-addresses in the form firstname at dyalog.com.

When members of the consulting team are not working for clients, the intention is that they will be members of the APL Tools Group at Dyalog, working on new tools for APL application development and helping create test suites for Dyalog APL. They will also support Richard Park, who joined us late in 2018, to work on the creation of training materials and tutorials for new users.

Once we have a better idea of the demand for consulting in North America, we expect to grow the team. Please let us know if you could use hired APL hands – in any territory! If we don’t have the resources ourselves, we may be able to find someone else.

Programming Language Implementors

Nathan comes to us with experience from a broad set of tools and programming languages. In addition to writing tools in APL, he will be a part-time member of the core development team, working on the APL interpreter and its interfaces in C, C#, JavaScript, Python and other languages. However, he won’t spend enough time on this to make up for the loss of Jay, who (like most managers at Dyalog) spent a significant amount of his time writing code.

Therefore, as described at https://www.dyalog.com/careers.htm, we are recruiting at least one C / C++ programmer to help us grow the core team.

A Busy – and Exciting Time

2019 is looking like an extremely busy year, with significant growth at Dyalog. As usual, our plan is to bring all the new (and old) hands to the Dyalog user meeting, which will be held in Elsinore, Denmark this year – September 8th to 12th. Details of the programme will soon start to appear at https://www.dyalog.com/user-meetings/dyalog19.htm. If you would like to present an APL-related experience to the user community, make proposals for new features of Dyalog products or suggest topics that you would like Dyalog to speak about at the user meeting, then please let us know as soon as possible!

Goodbye

At the end of May 2019 I am leaving Dyalog, so it seems like a good time to reflect on my time here and what I’ve learned from APL and the APL community.

When I joined Dyalog in 2010 I knew nothing about APL, so there was a really steep learning curve as I got to grips with both the language and its implementation. I was using some of my previous experience with compilers to improve the performance of the implementation, and thinking about ways to compile APL. This is a tough problem, and one that many people have worked on over the years (see for example Timothy Budd’s 1988 book An APL Compiler). My own ideas have shifted as I’ve gained more experience with APL and the way it is used. At first I thought “writing a compiler” was an obvious thing to do; now I think that hybrid compiler/interpreter techniques are much more promising, and Dyalog’s recent experiments with deferred execution and thunks are a good step in that direction.

At the same time, there was a lot of excitement around the APL language itself. Dyalog was working on APL#, a new .NET-based APL dialect (sadly abandoned as Microsoft’s own commitment to .NET waned). And Dyalog APL itself was starting to borrow more language features from the SharpAPL/J branch of the family tree, starting with the Rank operator and continuing over many years. This prompted me to delve more into the history of APL, to try to understand some of the fundamental differences between different implementations, so that we could reconcile those differences in Dyalog APL and provide, as far as possible, the best of both worlds. I think we’ve done pretty well in that, as evidenced by the fact that many APLers are happily using Rank, Key, function trains et al in an APL2-based language, something that seemed unthinkable a decade ago.

One of the most gratifying developments in the time I’ve been working with APL is the rapid growth of new APL implementations, open source projects and grass-roots enthusiasm. In particular, the open source movement has made it much easier for anyone with a good idea about language design to implement it, and share it with the world. We’ve seen a wide variety of new APLs and APL-inspired languages popping up over the years, ranging from full-featured to highly experimental, including but not limited to (in roughly the order I remember hearing about them): ELI, ngn/apl, GNU APL, Ivy, Aprildzaima/APL and APL\iv.

And speaking of new APLs, of course there is Co-dfns, a compiled APL implementation that tries to solve another tough problem: harnessing the power of GPUs and other massively parallel hardware in a way that makes it accessible to the end user. This is something that many people are trying to do, in a wide variety of languages, but as far as I can tell no-one has quite succeeded yet. The state of the art is still that, in order to get good performance, you need quite a lot of mechanical sympathy for the underlying hardware. But Co-dfns has come a long way, and if any language is well-suited to run on parallel array processors then surely it is APL!

This brings me on neatly to my next job: I’ll be working on compilers for GPUs, the parallel computers that render 3D graphics. They are closely related to their “general purpose” cousins the GPGPUs, which are used for pure number crunching, and to so-called tensor processing units (TPUs) that simulate neural networks for use in machine learning and artificial intelligence. “Tensor” here just means an array of arbitrary rank, or as we would say: an array. For programming TPUs there is a Python-based framework called TensorFlow. But, look closely at the APIs for some of the core TensorFlow libraries, and you’ll see operations like reshape, reverse and transpose, which are eerily similar to their APL equivalents. There truly is nothing new under the sun!

With fond regards to all APLers,
Jay.

Pool Stories: Fergus back on Form

It has been almost 2 years since our last post about Jo Fergus (daughter of our Customer Account Manager, Karen Shaw) the pool player who Dyalog have sponsored in the past. When we last updated you, Jo had just had her daughter, Remy, and made the difficult decision to take some time out from pool to concentrate on her new family.

On June 30th 2018 she married Tom Fergus, Remy’s Dad, in a small ceremony at Trowbridge County Hall and became Jo Fergus. The wedding came as a bit of a surprise to friends and family who thought they were attending a birthday lunch but the sun shone and a fantastic day was had by all. Jo started to miss playing pool and in 2018 she went to the Wiltshire county pool trials and was invited back into the Wiltshire Ladies A county team. After a successful season, in which they didn’t lose a match, the team has qualified for National County Finals taking place at the beginning of March. Jo has also joined a team playing in her local pool league.

In 2019 Jo took the decision to test herself and her pool skills and signed up to the 2019 IPA Ladies Tour which will see her play at 5 events across the UK. There are only 32 places on this Ladies tour and it will give Jo a chance to play against high level opposition, which she feels will improve her game. Ranking points will be awarded and the top 8 in the rankings after Tour 5 will automatically qualify for the first ever IPA Ladies Blackball Premier League.

Dyalog are proud to be sponsoring Jo for this tour and it is planned that all Ladies finals will be live streamed on freesports.tv so we have our fingers crossed she makes at least one final!

New York Dyalog Meetup

I am very pleased to announce the creation of the New York Dyalog APL Meetup group, details of which can be found online at https://www.meetup.com/New-York-Dyalog-APL-Meetup/. The meetup has been created and is organised by Paul Mansour, who is also sponsoring the venue for the inaugural meetup, scheduled for 6-9pm on Thursday February 7th, at Alley, 119 West 24th Street, New York. If you are interested in meeting APL users in the New York area, please join the Meetup group so that you will be notified of future events. Please sign up for events that you intend to attend so we know you are coming!

Meetup is a service used to organize online groups that host in-person events for people with similar interests, including programming languages. In addition to the New York group, there is also an APL Meetup group in Frankfurt which meets regularly. We welcome the creation of more local meetups! If you create one in your area, remember to inform us at Dyalog so that we can add a link from our event calendar, and arrange to stop by and speak when we are in your neighborhood!

The program for the meetup on February 7th is as follows:

6:00-6:30pm: Time for Networking

6:30-8:00: Morten Kromberg: New Ways of Working with APL

When you are busy solving problems, new technology can be an unwelcome distraction – but every now and again technologies appear which have the potential to make development, maintenance or distribution significantly easier. Morten will demonstrate some of the new ways of working with APL that have become available in the last few years, and also discuss likely features in the next couple of releases of Dyalog APL: 17.1 in Q2 of 2019 and 18.0 in 2020.

8:00-8:15 Short Break

8:15-9:00 Paul Mansour: Keeping it Simple – A Git Workflow for APLers.

Abstract: Git is great, but the newcomer can easily drown in a sea of commands and options. Git doesn’t tell you when or why to branch, when or why to merge or rebase, how to version your project or prepare a release. AcreFlow is a radically simplified Git workflow that answers these questions. It is implemented in Dyalog APL so you can branch, commit, and put out new versions directly from the APL session.