Monday 22 September at Dyalog ’14

This was originally posted to Catalyst PR’s Facebook page and is reproduced here to make it accessible to people who don’t have Facebook accounts.

For information on the presentations at Dyalog ’14, see

After having spent yesterday in workshops with lots of hands-on examples, today was the kick off for the user meeting proper. The last time Dyalog held a global user meeting in the UK was in 2003, and 2014 has given us an all-time record attendance in terms of delegates.


My posts from Dyalog ’14 will focus on extracts from User Presentations, and the first presentation I would like to highlight is There’s DNA Everywhere – an Opportunity for APL by Charles Brenner PhD (

Charles Brenner

Charles Brenner

Charles describes himself as consulting in forensic mathematics and his DNA-View software programme is used by 100 laboratories in every continent (except Antarctica) for academic activities in mathematics, biostatistics and various aspects of population genetics.

Charles used the example of gun crime to illustrate: “Use a gun in a crime and, thanks to recent biochemical advances, you’ll likely leave enough DNA to be detected. However, often the gun has also been handled by others and, consequently, the analysis of DNA mixtures has become increasingly important. The market for good DNA mixture software, therefore, has led me to recent and rapid progress in my long-delayed venture to convert my forensic DNA interpretation work into the modern world of Dyalog. Many people enjoy the scientific aspects of the story and I enjoy relating how, in my view, this project exemplifies the thesis that APL is a tool of thought. I credit APL with leading me to elegant and, therefore, very fast and flexible solutions to a problem for which the competing solutions are lumbered by complicated statistical and Monte Carlo methods.”

Elegance and simplicity lead to several concrete benefits. The first is conceptual development.

Slide from talk I01

Slide from talk I01

“I made notes in APL as solutions to various aspects of the problem occurred to me last year. These brief APL notes ensured that the ideas really made sense, that they worked together and that I would not forget them. The brevity also revealed a simple but important point that others had overlooked: nesting the computation loops in the right order saves orders of magnitude in computation time. For another, execution speed makes it much easier to see the forest in many ways – testing, developing, designing. Competing programs take hours to find a partial answer, then having worked that hard succumb to a natural tendency to call it a day. The APL “DNA-VIEW Mixture Solution” program does the same in seconds, which makes it much easier to think through to the fact that there’s a lot more work to do before the result is logically defensible.” Charles explained.

At the end of 2013 Charles implemented the Mixture Solution and tested it on examples including a set of five proficiency exercises created at the National Institute of Standards and Technology. More than one hundred entrants, including the leading competition, had contributed analyses. One of the competitors (supported by years and millions of dollars of government grants) got four problems right and came close on the fifth, viewing it as a three person mixture although in fact it was four. Mixture Solution alone correctly analysed all five – as a bonus it correctly diagnosed one suspect as a mixed race person.

“We APLers try to be modest but it’s not always easy. Sometimes there’s just not much to be modest about!” Charles stated.

Having now reached a point where he has developed a superior, fast, elegant software solution to a massively complex DNA Mixture analysis problem, Charles ended the presentation with a plea for help. First and foremost he is looking for someone who can help carry on with the further software development using Dyalog. This could be a fantastic opportunity for someone with an interest in forensics, maths, computer science and software programming in Dyalog – “living in the Bay area”.


The Finnish Pension System is currently undergoing a reform and there is a strong need to microsimulate various parameters to help make decisions regarding future pension schemes.

Slide from Talk U02

Slide from Talk U02

Heikki Tikanmäki from the Finnish Centre for Pensions gave us a presentation of ELSI – the Pension microsimulation model that has been built in the Finnish Centre for Pensions using Dyalog.

ELSI models the statutory pension system of Finland. It is used for reform analysis and long-term projections. In a typical simulation run, they model 500,000 individuals over 70 years.

By using Dyalog not only do they retain compatibility with older macromodels, population and employment projections, they also gain flexibility for modelling various proposals for a new pension scheme.

The model is built in modules with each module running as independently as possible. All the simulation data are held in component files. Depending on the complexity of each module, it takes 10-40 minutes to run using version 12.1 – without doing any performance optimisation.

Heikki told us that by upgrading to Dyalog version 14.0 the model runs 20% faster and the simulation data are now only taking up 16% space in comparison to the 5 GB size they previously had.

Future plans include implementing the use of parallel processors for the computational tasks in ELSI, and at that point in time he expects at least a further 50% reduction in the time it takes to run the entire microsimulation model.

Postcard from Dyalog ’14 – Monday

Delegates by country

Delegates by country (click to expand)

Monday opened with registration for what is the biggest Dyalog user meeting on record – it could have been even bigger but we had to turn some people away because there was no more room available. In total there are 126 attendees (even more that when we went to print with the programme!) made up from 95 delegates, 12 spouses and partners, and 19 Dyalog employees from around the world as shown.

The Conference Hall

The Conference Hall

Discussion Point: DNA Analysis

Charles Brenner performs forensic analysis of DNA and DNA mixtures – an intensive mathematical process. Others use complicated statistical and Monte Carlo methods but by using APL as his tool of thought, Charles has devised techniques which are more accurate and many times faster than competing applications supported by multi-million dollar funding.

Discussion Point: A Different Kind of Selfie

In today’s session “The Tuning Pipeline”, Roger Hui noted that for the index-of family of functions, a faster algorithm is often possible if the left and right arguments are “the same”, up to a factor of two. “The same” means the that the values have identical references so that they can be compared with a simple and quick check. For example, x⍳x is a selfie but x⍳0+x or x⍳(⍴x)⍴x are not.

For example, for the inverted-table index-of 8⌶,

      u←(' ',⎕a,⎕d)[?8e4 30⍴37]
      cmpx 'p(8⌶)p' 'p(8⌶)q'
  p(8⌶)p → 3.98E¯3 |    0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕                      
  p(8⌶)q → 8.72E¯3 | +118% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

Selfies occur in less obvious places – when finding uniques (∪x), in the key operator, and in x∧.=⍉y as well as in x⍳x and ⍳⍨x. You can try this yourself for various datatypes. A selfie which is not faster is an opportunity for a performance improvement.

Discussion Point: Sorting is Faster Than Grading

Also in “The Tuning Pipeline” session, Roger Hui and Kimmo Kekäläinen looked at some of the recent performance improvements in Dyalog. It has long had idioms for sorting ({⍵[⍋⍵]}, {⍵[⍋⍵;]}, etc.) Interestingly, for some common datatypes, sorting is faster than grading. You can try this yourself for various datatypes:

      cmpx '{⍵[⍋⍵]}x' '⍋x' ⊣ x←?1e6⍴2e9
  {⍵[⍋⍵]}x → 3.23E¯2 |    0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕                         
* ⍋x       → 8.34E¯2 | +158% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

The point is written up in this essay on the J website. The webpage uses J but the explanation applies to APL (or any other language). Quoting from the webpage: “Grade needs to keep track of the argument array and the list of indices. Sort just needs to keep track of the first. … When the argument items can be manipulated efficiently, as would be the case when the items are machine units (1, 2, 4, or 8 bytes), then sort can eschew a separate index array.”

John Scholes’ “Distractions” – An Objective Review

An awed silence gripped the room as John unveiled ground-breaking techniques for minimising life’s distractions and maximising programming productivity. It is too early to say how far-reaching this approach will turn out be in real life situations and whether it will affect The Global Economy as a whole. John appeared to have all categories of distraction covered. Let’s see. Like Woodstock ’69 and The Isle of Wight ’70, in years to come you may be able to say “I was there”.

John Scholes

John Scholes

John Scholes

John Scholes

And Also …

Some of the many other things we saw and heard today:

  • Dyalog has taken on two new employees since Dyalog ’13
  • John Daintree can’t take selfies but embraces high resolution touch devices
  • Data files used in Finnish pension microsimulations reduced to one sixth of their size when component file compression was enabled
  • MyDyalog launched live at the user meeting
  • Fiona wants to promote your APL application in a banner on the Dyalog homepage
  • Morten computed Mandlebrot set images, performing the calculation in parallel with isolates running on servers in Holland and Hong Kong


Tomorrow’s schedule features five user presentations and four Dyalog presentations covering the themes of code management and reuse, performance, presentation tools and cryptography. In the evening Morten will demonstrate the progress he has made with the ‘bots (something that should be very familiar to regular readers of the Dyalog blog!).

Postcard from Dyalog ’14 – Sunday

Today at Dyalog ’14 was a day of Technical Workshops: six half-day sessions and one all-day session were held, and there will be more at the end of the user meeting on Thursday.

JD's Introduction to DWA workshop

JD’s Introduction to DWA workshop

This evening Gitte will informally welcome the attendees before Vibeke introduces the scavenger hunt and Karen and Jason run a traditional English pub quiz so that delegates can get to know each other and the host country a bit more.

Getting Involved

This year we plan to keep you more up to date than ever before with what is happening at the user meeting. Programme updates and supporting materials will be made available on our website. There will be blog posts each day and we are using the Twitter hashtag #dyalog14 on our tweets – we encourage everyone to join in with us whether in Eastbourne or not.


For Team Dyalog, the start of the user meeting is the culmination of preparations spanning weeks or even months. There have been some obstacles along the way – not least the fire on the pier, which meant a replacement banquet dinner venue had to be hastily found – but the presentations are ready and everything kicks off in the morning. We are looking forward to a busy but enjoyable week!

Jason repairing fire damaged robots

Jason repairing fire damaged robots *



Setting up the conference room

Setting up the conference room

No one is saying what these are for

No one is saying what these are for

* See this blog post


Dyalog ’14 formally begins tomorrow. We expect a busy day commencing with registration and followed by ten presentations in total, starting with a formal welcome from Gitte. As has become the custom, there will be a mixture of presentations from Dyalog Ltd, guests and invited speakers throughout the week. Monday traditionally features Dyalog presentations and this year is no exception – but we will also have Charles Bremner and Heikki Tikanmäki talking about forensic DNA analysis and pension modelling respectively. Two years ago, by popular request, we abandoned the practice of running presentations in parallel (extending the programme by a day to allow this) and all of the presentations will run sequentially again this year.

There are planned evening activities for each of the conference nights. After dinner tomorrow, John Scholes will present his observations on the subject of “distractions” – those who have attended his evening presentations before will know they can be highly entertaining!

Postcard from Dyalog ’14 – Saturday


Welcome to the first of several postcards from Dyalog ’14 in Eastbourne. The last Dyalog conference/user meeting held in the United Kingdom was at Horsley near Guildford back in 2003 and we’re looking forward to holding it on what is, for a lot of us, home ground. Eastbourne is a resort on England’s south coast known for its promenade and pier (of which more later!), its hotels, guest houses and, by no means least, its conference centres. As a bonus, we should see some decent seaside weather – we can’t promise the Florida sunshine of Dyalog ’13 but the UK is currently having a much better than average late summer; the weather forecasters say it should continue and that we’re heading for one of the driest Septembers on record so let’s hope they’ve got it right!

Eastbourne weather forecast

All but four Dyalog staff have come to Eastbourne. We’re looking forward to sharing what we’ve been working on, listening to our users’ presentations and meeting attendees old and new – this is set to be the biggest conference we have ever held with 126 registrations in total (two more than it says in the programme due to two last-minute registrations after we went to press!).

Coming Up…

The programme kicks off tomorrow with Technical Workshops all day followed by three and a half days of presentations by Dyalog staff, users and invited guests, finishing on Thursday afternoon with the remaining Technical Workshops. We’ll report on some of the highlights of these presentations over the coming week. We’re making recordings of all the presentations and some of the workshops, and will be putting them all up on the Dyalog website over the coming weeks.

We do, of course, also have our “Viking Challenge” on Wednesday as well as daily evening entertainment, culminating in the banquet on Wednesday night. Originally we had the dining suite at the end of Eastbourne Pier booked for the banquet, but, as you may have seen, Eastbourne Pier suffered a major fire at the end of July and is now closed for rebuilding. Fortunately we have been able to secure The Grand Hotel as an alternate venue, and all is back on track.

All in all we anticipate a very busy – but enjoyable and productive – week and hope to summarise the essence of it here; check back daily for reports of events!

Turning to a Heading with an MPU-9150

As the odour of fried electronics dissipates in the air, I’m unexpectedly afforded the opportunity to write this blog post a day or two earlier than planned. The on-board compass was exhibiting significant deviation, so I consulted my nephew Thorbjørn Christensen at DTU-Space. Thorbjørn makes a living designing magnetometers for space agencies around the world, and he suggested I use a ribbon cable to move the IMU away from the bot as a first step towards understanding the issue. He did warn me to be very careful when attaching it. I still don’t understand what I did wrong, but the evidence that I screwed up is pretty definitive: there are burn marks around every single connector and the IMU chip looks a bit soft in the middle. Most importantly, it no longer reports for duty on the Raspberry Pi’s I2C bus!

A fried IMU-9150

A fried IMU-9150 (click to enlarge)

The good news (apart from the fact that the Raspberry Pi, the Arduino and all other ‘bot components seem to have survived) is perhaps that, since I cannot play with the compass until I have a replacement, I found the time to write about where I am at the moment…and take some time out to work on my presentations for Dyalog ’14 (for those who managed to register before the user meeting in Eastbourne sold out, I still hope to do a ‘bot presentation on the evening of Tuesday September 23rd).

Introducing the MPU-9150

For the last few weeks, I have been using my “bot time” to play with the MPU-9150 breakout board that is attached to our ‘bot. The MPU-9150 is more or less identical to the 6050 that we were using earlier this summer to make gyro-controlled turns but also includes a magnetic compass, which will allow us to know the direction that we are heading in – making it a 9 degree of freedom inertial measurement unit, or “9-dof IMU” (9 = 3 gyro axes + 3 accelerometer axes + 3 compass axes).


I was extremely happy to discover that Richard Barnett had shared his RTIMULib on GitHub. RTIMULib is a Linux library that produces “Kalman-filtered quaternion or Euler angle pose data” based on data from a 9-dof IMU. Jay Foad at Dyalog was quickly able to provide me with a couple of additional C functions designed to be easy to call from Dyalog APL. Jay’s fork of RTIMULib is also available on GitHub. Since we can assume that we are driving on a flat surface, I have just been using two items from the wealth of information provided by this library: the current rate of rotation and the current compass direction – in both cases around the vertical (or “z”) axis.

My First Compass-Controlled Rotation

Due to the fried IMU, I am unable to post a video recording; fortunately, at the moment where I toasted the chip, the Dyalog session output still contained output data from the last run of the function I am working on, which aims to provide smooth rotation to a selected compass heading. In the chart below, the red line tracks the speed of rotation and shows that the “smooth” part still needs work – the speed oscillates quite violently between 60 and 150 degrees per second. The blue line shows the number of degrees that the ‘bot still needs to turn to reach the desired heading. The dotted red line is slightly mislabeled: rather than acceleration, it actually shows the power applied to the motors via a digital-to-analog converter that accepts values between 0 and 255, producing between 0 and 5v of voltage for the DC motors:

Rotating to a compass heading


  • From 0 to 0.4 seconds, we slowly increase power until we detect that rotation has started. We note the power setting required to start rotation.
  • 0.4 to 0.7 (ish), we detect acceleration and hold the throttle steady (increasing it very slightly a couple of times when acceleration appears to stop).
  • 0.7 to 1.1 seconds, cruise mode: whenever speed is above the target of 100 degrees/sec, we idle. When speed is below 100, we re-apply power at a level that was sufficient to start the rotation.
  • 1.1 to 1.4 seconds: We are less than 30 degrees from target, and the target velocity and thus power is reduced proportional to the remaining distance (at 15 degrees, we are down to 50 deg/sec)
  • 1.4 to 1.6 seconds: We detect that we rotated too far, and slam on the brakes, coming to a stop about 10 degrees from target.
  • 1.6 to 2.3 seconds: since we are less than 10 degrees from target, we enter “step” mode, giving very brief bursts of power at “start rotation” level, idling, and watching the heading until we reach the goal.
  • 2.4 seconds: We count the bursts and, after 10, we double the power setting to avoid getting bogged down (you cannot see the bursts in the chart, it only records the power level used for each one). A little more patience would have been good, this probably means we overshot by a bit.

The “Heading” Function

Achieving anything resembling smooth movement is still some way away, mostly due to the fact that motor response to a given input voltage varies enormously from motor to motor and between different applications to the same motor. The strategy described above is implemented by a function named “Heading”, which can be found in the file mpu9150.dyalog in the APLBot folder on GitHub. An obvious improvement would be to have the robot self-calibrate itself somehow and add a model of how fast rotation speeds up and slows down (based on, and constantly adjusted with, observed data), in order to dampen the oscillations. I intend to return to this when I have a new IMU (and my other user meeting presentations are under control).

This has turned out to be a lot harder than I imagined: suggestions are very welcome – please leave comments if you have a good idea!