Dyalog Ltd's CEO opens the user meeting.
The Road Map provides a high-level view of recently completed projects and active development, and a declaration of intent with respect to the areas where Dyalog Ltd is likely to invest significant energy in the next few years.
Keywords in 2015: cross-platform application development, new development tools, and (as always) performance.
The RIDE (Remote Integrated Development Environment) is a graphical front end that can be used to develop and maintain Dyalog applications, irrespective of the computer that the Dyalog engine is running on. Nick explains how you can make the most of RIDE's latest features and enhance your coding comfort and efficiency by customising the default keystrokes, characters, and colours.
Recent and future versions of Dyalog have benefitted from several implementation techniques, including:
Roger and Jay will discuss these techniques and the impact they have on the performance of the interpreter.
D05: User Commands Update summary | video (20 mins)
Adám Brudzewsky & Dan Baronet
This presentation concentrates on the latest changes to user commands. Particular emphasis is given to the new ]CopyReg command (introduced at version 2.01 with Dyalog version 14.1), which enables a Microsoft Windows user to copy configurations from one version of Dyalog to another.
Andy highlights some of the new features that have been added to MyDyalog or will be available in the near future, and explains some of the changes to the way in which updates are supplied, etc. Much of what he describes in this talk is Jason Rivers' work (Andy is presenting it since Jason is tied up with AV duties this week); if you have any feedback on MyDyalog then please let Jason know.
In 2015, Dyalog released the first version of Dyalog APL for Mac OS, and other platforms will be targeted in the near future. Cloud deployment provides opportunities to deliver web applications on both Microsoft Windows and Linux. An important challenge is to find a mechanism that can make it straightforward to develop user interfaces to run on all these platforms with a minimum of effort.
Popular modern programming languages benefit from access to good quality, community-built, code libraries. However, there has been no great tradition of this in the APL community; this is something that Dyalog wishes to address.
The interpreter can exchange data and workspaces between different platforms with ease – but has also evolved in ways that make it difficult to distribute a single, portable, code library. For example, filename extensions are different on Microsoft Windows and non-Windows platforms, and code that wants to access the underlying host file system often has to invoke non-portable host commands to do so.
Dyalog version 14.1 addressed some of the problems of filename extensions. Version 15.0 is expected to take this further and also introduce new system functions to allow greater access to the underlying file system. Richard explains the changes that have been made and those that are being planned and the thinking behind them.
Following a brief review of Dyalog's social media, Fiona turns her attention to one of the main channels through which we communicate with the world – the Dyalog blog. Contributed to by all members of Team Dyalog, posts range from the highly intellectual to the rather more frivolous and cover topics as diverse as explorations of language elements, recollections of why APL evolved in the way it did, personal musings on the future of APL, solutions to previous student competition questions, travelogues and (of course!) the Dyabot. Fiona concentrates on three particular blog posts from the last 12 months and asks the authors of those posts to summarise and take questions/feedback on their musings. Up for discussion are:
Dyalog intends to develop new tools for application development. A central idea is the notion of a shared definition of a "Dyalog Project": a mechanism for declaring the source code components that an application is built from and build directives to build and deploy different variants or sub-sets of the application, for different uses or target platforms. Future tools are likely to revolve around such "project definitions", which need to be sufficiently flexible to allow new users to ask for "A New Dyalog Web Application" and have it up and running in minutes, and still describe legacy applications well enough to allow selected tools to be brought to bear on them.
Keywords: Source Code Management, Dependency Management, Building and Deploying applications.
Databinding has been extended to support binding to matrices. John demonstrates this, along with more ideas for databinding in the future.
An External Workspace is a workspace saved in a new format that allows an application to page code and data into the active workspace on demand, rather than loading the entire workspace when the application starts. The files are memory mapped, which means that processes running on the same machine can share the memory used by the parts of the workspace which have been paged in, and the file is only read by the first process that uses any particular function or variable.
SimCorp Dimension is a rapidly growing application. At run time, code is usually loaded from a network drive. The time required to load the workspace and subsequently to page code in from component files is also steadily growing. Details of the implementation of External Workspaces at SimCorp will be discussed – including some timings.
D14: Waste Time with John Daintree summary | video (47 mins)
John Daintree, Chief Architect
Get your geek on with John Daintree as he has a devil of a time with JSON and Databinding. May contain images of partial nudity...
D15: Compilation and Bytecode Execution summary | slides (165 KB)
Dyalog version 14.0 included an experimental compiler for APL functions, allowing you to compile functions ahead of time and run them with an optimised bytecode execution engine. Dyalog version 14.1 greatly extended the support for compiling tradfns and for functions that reference other global functions, variables and constants. Jay will demonstrate these features and describe further improvements that are planned for the future.
Three operators are presented to stimulate discussion and feedback from the user community. Roger opens with:
John then looks at merge, exploring some suggestions for the functional merging of arrays as alternatives to the procedural mechanism of indexed and selective assignment.
SharpLeaf is a typesetting and tabulation engine that allows automated production of high-quality reports using a simple yet powerful API. It is the replacement for NewLeaf, and will be bundled within the SharpPlot package. Nic takes us through why and how to use it.
Last year Andy spoke about some of the new(ish) or lesser-known little features of Dyalog that are buried deep in the documentation. There's plenty that he didn't mention, and even more with Dyalog version 14.1, so he's back with the second in a potentially never-ending series of conversations...
As ever, let us know if there are any features that you think would make Dyalog easier to use.
The calculation of premiums in German health insurance is based on some fundamental actuarial tables, an expected long term average interest and the premise that the individual premium shall not depend on aging itself, but only on the age at contract date. Once tight regulations are also taken into account, the business model of German health insurance is built around (huge) benefit reserves and security margins.
Its surplus stems mainly from interest on reserves (above the technical interest rate) and security margins. German law requires insurance companies to give back to the policyholders at least 80% of the entire surplus within 3 years. This has to be done after agreement with an independent trustee.
One of the two main ways to use the accumulated surplus is to cap premium increases during recalculation of premiums. Simulating this is complicated because capping is highly individual and needs the recalculation of all (individual) premiums as a pre-requisite. The calculations are straightforward but consist of a very large number of simple arithmetic operations.
The main challenge is to reliably achieve high accuracy, while keeping computation time in the limits needed for high flexibility. Furthermore it may not be obvious at all what a "fair" distribution of funds means and what the right capping scheme may look like, so a substantial amount of output information is needed.
Using APL, different capping models can be simulated on the basis of one premium recalculation simulation, using – as far as possible – 4-byte integer main arrays and pure vector and matrix calculations. The resulting computation times allow the simulation of many models. Furthermore, the costs of a model complete with error margins can be estimated within seconds.
To maximise our potential as individuals, teams and (ultimately) organisations, we have to take a step back from all of the enabling technologies, processes and tools that surround us and look inwards instead. In this presentation, Gianfranco shows us how to do this by touching on three subjects that start with "you and me" at the centre – Communication, Transparency and Practice.
For those who need to move data back and forth between APL and Microsoft Excel, the options are growing. With Microsoft Office 2007, Excel and other Office documents adopted an open, XML-based file format. From an APL perspective, XML is nothing more than a delimited string of text – and we know how to handle such things in APL (in addition to which we have ⎕XML). Several object-oriented, .NET-driven class libraries exist for manipulating *.xlsx files without the need for Excel itself. These include Syncfusion's XLsIO, which is now provided by, and integrates easily with, Dyalog. Office Automation, using COM techniques, still works but has its limitations (and, of course, still requires Excel to be installed). The newer XML-based tools also have limitations, some of which can be overcome with familiar APL array-oriented magic.
U04: How to Save an Italian School with APL summary | video (34 mins)
Roberto Minervini, Liceo Classico S. M. Legnani (Italy) with Arianna Greco, Samuele Luigi Di Gioia, Kariman Ahmed & Silvana Maria Belén Colman Cabrera
Could APL be a good approach to teaching maths/programming in a secondary school?
Roberto tried to answer this question, and he and some of his students tell the story of the APL course used at the school over the last few months. The scene is set with a description of how maths/programming was taught in the Liceo Classico S.M. Legnani secondary school before APL was involved, and the resultant problems. They then explore how maths (both teaching it and learning it) can be completely revolutionised with the introduction of APL.
Over the past 3 to 4, years Optima has been developing COSMOS; a tool designed to visualise data. Many of you will have seen COSMOS demonstrated in the past and may recall the way data is displayed as interlinking nodes. In this presentation Paul will discuss some of the more recent changes that have occurred to the product and how it is evolving in response to clinical need.
There are currently 800 cancer-related drugs awaiting clinical trials in the US but not enough people to trial them on. Even if there were, the identification of the right person for any particular drug is not trivial. Data is now being collected to assist in this process and we have an opportunity to make a real difference to the Clinical Study model; how do we identify and track suitable patients and how can we see what is, or is not, working in those trials?
What has come to light is that the requirements of the medical industry vary from company to company, drug to drug, and any analytical platform needs to be able to respond to those variations very fast. APL has, once again, come to the rescue and given us the ability to produce a platform that can be moulded to fit those needs quickly and easily.
However, all this does not come without its problems. As part of this talk, Paul will show the solutions to some of these problems as well as some problems that are yet to be answered...
U06: Ants, AJAX and APL summary | video (31 mins)
Ray Cannon, Cannon Computer Consultancy Ltd (U.K.)
Ray demonstrates "Ray Ant World", a visualisation of a match between two species of ants in a world containing two anthills, some food sources, and several obstacles, running under the rules laid out in the 2004 ICFP Programming Contest – "Dinner with Ambiants".
Phil explains why he thinks there's a big hole in APL notation and gives his own suggestions as to how to plug it.
U08: Data-bound Dashboard summary | video (16 mins)
Alexey Miroshnikov, InfoStroy Ltd (Russia)
Alexey demonstrates how a few lines of code using new features of Dyalog APL allow him to provide data visualisation of complex financial data by embedding the DevExpress "Dashboard" object into GAMA, InfoStroy's asset management product.
The seventh annual International APL Programming Contest took place earlier this year. Brian introduces the contest and grand prize winner and Walt Syzonenko (Fiserv) presents her with her certificate in the Prize Ceremony.
Arianna Locatelli is the grand prize winner of the seventh annual International APL Programming Contest. Following the prize ceremony, Arianna will describe her experiences of Dyalog and tell us how she managed to master enough of it to win the competition.
U10: Abstract Expressionism for Parallel Performance summary | slides (250 KB)
Robert Bernecky, Snake Island Research (Canada)
Programming with abstract, mathematical expressions offers benefits including terser programs, easier communication of algorithms, ability to prove theorems about algorithms, increased parallelism, and improved programming productivity. Common belief is that higher levels of abstraction imply a larger semantic gap between the user and computer and, therefore, typically slower execution, whether sequential or parallel. In recent years, domain-specific languages have been shown to close this gap through sophisticated optimisations benefitting from domain-specific knowledge.
In this presentation, Robert demonstrates that the semantic gap can also be closed for non-domain-specific functional array languages, without requiring embedding of language-specific semantic knowledge into the compiler tool chain. He presents a simple example of APL-style SaC programs, compiled into C-code, that outperform equivalent C programs in both sequential and parallel (OpenMP) environments.
Robert also offer insights into abstract expressionist programming by comparing the characteristics and performance of a numerical relaxation benchmark written in C99, C99 with OpenMP directives, scheduling code, and pragmas, and in SaC, a functional array language. He compares three algorithmic styles: if/then/else, hand-optimised loop splitting, and an abstract, functional style whose roots lie in APL. SaC algorithms are shown to match or outperform serial C, and the hand-optimised and abstract SaC styles generate identical code, and so have identical performance. Furthermore, parallel SaC variants also outperform the best OpenMP C variant by up to a third, with no SaC source code modifications. Preserving an algorithm's abstract expression during optimisation opens the door to generation of radically different code for different architectures.
U11: Using Co-dfns to Accelerate APL Code summary
Aaron Hsu, Indiana University (U.S.A.)
This live coding session will cover the expected capabilities of version 1 of the Co-dfns compiler project and demonstrate how to begin testing and experimenting with the compiler using the publicly-available pre-release versions. It will demonstrate how to take an existing APL program and integrate Co-dfns into it to best effect to assist in optimising the execution time of the code. It will also cover the currently-supported hardware platforms and give some guidance on choosing the right hardware platform to maximise performance. This session is designed to provide a practical introduction to using Co-dfns with their own code, from initial installation to final deployment of Co dfns compiled code.
U12: Troubles with Strange Data Structures and Database Growth summary | slides (625 KB)
Francesco Garue, APL Italiana (Italy)
Whenever software produces a number as the result of a computation, European insurance authorities require a lot of additional data to be saved so that the computation can be "reasonably" explained. If the data structure is inappropriate and/or redundant, then database growth can spiral out of control. Some of APL Italiana's customers began claiming that the volume of additional data was growing by an order of magnitude faster than the rest of the database and their disks were filling up, so something needed to be done – Francesco explains what was done both to change a particular data structure without re-writing the whole code base and to compress the data (no DCF file compression was available at that time).
U13: TamStat, a Statistical Package summary | slides_SM (660 KB) | slides_JB (7.2 MB) | video (42 mins)
Steve Mansour, The Carlisle Group and University of Scranton (U.S.A.) & Jerry Brennan, Jerry M Brennan PhD Inc (U.S.A.)
TamStat is a statistical package written in Dyalog. What distinguishes it from other statistical programs such as R, Minitab and Excel are its natural use of arrays, operators and namespaces. While users have the option of using the APL character set, there are ASCII cover functions that allow them to use a traditional keyboard if they so desire. There is also an option to call R to do some calculations using the RConnect workspace. Recent developments include a function to import data from Excel, a Reference Card, a User Guide and a web interface using MiServer. The MiServer interface includes plotting routines, tabbed interactive tutorials and videos for learning statistics, using TamStat and programming in APL.
SA1: Building a WPF Application: Part I summary
Michael Hughes (from MJH Software Services Ltd.) asst. John Daintree
At the end of the two "Building a WPF Application" workshops, participants will have a small, working WPF application to take away, along with a practical understanding of using WPF with Dyalog.
The workshop is practical with a heavy reliance on "doing" rather than just "listening". It is divided into two halves:
Particular emphasis will be given to:
Part I comprises three sessions:
SA2: Building a Web Application with MiServer: Part I summary
Brian Becker (Tools Group Manager) asst. Bjørn Christensen
SA3: Parallel Programming with Futures and Isolates: Part I summary
Morten Kromberg (CTO) asst. Adám Brudzewsky
Futures and isolates were introduced in Dyalog version 14.0. These language features are designed to make it straightforward to harness all the cores that your computer (or cluster of computers) has available, whether you are writing a brand new application or need to speed up legacy code. In this hands-on workshop we will introduce these new language features, with plenty of time for exercises to allow you to immediately practise new-found skills.
SA4: Recent Language Features in Practice summary | slides (820 KB)
Dan Baronet asst. Nick Nickolov
Familiarise yourself with some of Dyalog's language features, focusing on more recent ones like the Rank (⍤) and Key (⌸) operators introduced in Dyalog version 14.0.
Aimed primarily at those people who are planning to use (or simply want to improve their understanding of) these features, several problems of varying difficulty will be solved as we explore the possibilities that recent language features bring to Dyalog.
SP1: Building a WPF Application: Part II summary
Michael Hughes (from MJH Software Services Ltd.) asst. Dan Baronet
At the end of the two "Building a WPF Application" workshops, participants will have a small, working WPF application to take away, along with a practical understanding of using WPF and Dyalog.
The workshop is practical with a heavy reliance on "doing" rather than just "listening". It is divided into two halves. Part II is a hands-on session in which knowledge from Part I will be assumed.
Part II comprises three sessions:
SP2: Building a Web Application with MiServer: Part II summary
Brian Becker (Tools Group Manager) asst. Bjørn Christensen
Following on from the morning's introduction (workshop SA2), in this hands-on workshop we'll build a web application – either transforming an application you bring to run under MiServer, or using a sample application we provide. We will examine different strategies for deploying applications with MiServer including stand-alone, running within the corporate infrastructure or as a web service. Topics covered will include creating look and feel with styling and using widgets to produce rich user experiences.
SP3: Parallel Programming with Futures and Isolates: Part II summary
Morten Kromberg (CTO) asst. Adám Brudzewsky
Following on from the morning's introduction (workshop SA3), this workshop provides time and space for actual application development. We will analyse a worked example in depth, but the bulk of the afternoon will be set aside for you to work on your own application with help from experts and the other workshop participants (if you are willing to display your code). Bring your own application along, or build one during the afternoon session (if you don't have one, some exercises will be provided).
SP4: "The Doctor Is In" summary
Andy Shiers (COO), John Daintree (Chief Architect) & friends
Pop in to the APL surgery!
In previous user meetings you have always been welcome to talk to any of the Dyalog team about your APL/Dyalog problems (and ideas). That hasn't changed this year (and is unlikely to in future user meetings either!) but we thought that this year we'd also put some specific time aside for this. In this workshop Andy, John and others are available without other distractions. There's no fixed plan, but we would like to get to talk to as many people as possible, so if you have a very involved issue we might ask to continue the discussion at some other time.
NOTE: This session will continue after the end of the other workshops, so don't worry if you want to attend one of them…go to the other workshop and either pop in to see us afterwards or we can always try to find time to talk during the week.
We don't guarantee to solve your issue there and then, but we can try – and that can make a big difference!
TP1: Excel Mining – Exploiting Microsoft Excel's Open XML Structure summary
Richard Procter (from APL Borealis) asst. John Daintree
This workshop covers a range of techniques that can be used to move data back and forth between an APL workspace and Microsoft Excel. Starting with a brief review of the former standard methods, including *.csv files and COM automation, the bulk of the session will be spent exploring Excel's open XML-based structures. We will use some readily-available class libraries as part of identifying the best practices for converting Excel spreadsheets to APL arrays and vice versa; these include the Syncfusion XLsIO library (included with recent versions of Dyalog); a handy utility suite based on that library, and another approach using Microsoft's Open XML SDK. These object-oriented tools work well but can have limitations in some cases. We will learn how to deal with these limitations by putting the array-oriented speed and simplicity of APL to work.
A basic understanding of the use of .NET techniques in APL will be helpful but is not a requirement.
TP2: Accessing External Data from Dyalog summary | slides (1.8 MB)
Dan Baronet asst. Adám Brudzewsky
Not all data is held within workspaces – often it is contained within external data files such as Microsoft Excel files, CSV files or XML files. This workshop reviews the different ways in which data outside the workspace can be accessed. There will be several examples and problems to be solved using each of the aforementioned external data files as well as JSON files, Web services, databases and native and component files.
TP3: Creating and Managing Dyalog-based Services summary
Brian Becker (Tools Group Manager) asst. Andy Shiers
In this workshop we'll look at setting up Dyalog applications to run as Microsoft Windows services and using daemons under Linux. We'll explore a variety of topics including architectural changes for running as a service, managing services, and usage and error logging. We'll also delve into performance and load balancing by running Dyalog applications using multiple processes, either on a single platform or across multiple platforms. A sample service will be provided for you to deploy and experiment with.
TP4: Writing Efficient Code in Dyalog summary | materials (30 KB)
Roger Hui asst. Nick Nickolov
This workshop is divided into two sections, each looking at different aspects of coding efficiency.
Part I – Performance Tips: A review of performance tips and rules of thumb useful for writing efficient code.
Part II – Special Code: Many primitives contain special code for certain arguments to effect time and/or space savings not available to general arguments; moreover, some phrases are recognised and are supported by special code (these are in addition to idioms). We explore this special code with emphasis on the newer constructs offered by Dyalog.
Eugene Ying: CSV_processing.pdf
UNIX® is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited.
Windows® is a registered trademark of Microsoft Corporation in the United States and other countries.
Oracle® and Java™ are registered trademarks of Oracle and/or its affiliates.
Mac OS® and OS X® (operating system software) are trademarks of Apple Inc., registered in the U.S. and other countries.
All other trademarks are the property of their respective owners.