Parallel Programming with Futures/Isolates

DSC05195Morten Kromberg went right to it with his well-prepared workshop on the prototype features that were introduced in version 14.0 of Dyalog APL. The features are still partly modelled in APL to allow the first wave of users to pass judgement on the design. Throughout the workshop, we were asked for feedback, to ensure that the new primitives will be as well implemented as possible.

Everyone is encouraged to apply the new tools in practice so they can be adjusted and possibly get expanded functionality based on feedback from actual use-cases.

There is no denying the the obvious performance gains that Morten demonstrated: He was able to let the computer waste 9 idle seconds in only slightly more than 3 seconds!

Joking aside, one practical example that showed how useful parallel processing can be, was collating character counts from online news sites. Because of latency in loading these pages over the internet, starting many parallel threads that each requested a page greatly reduced the wait time. DSC05197 One workshop participant volunteered a piece of code that needed to be parallelized got up and pair programmed this with immediately achieving significant performance gains.

The idea of Futures is allowing the continuation of other work before an asynchronously launched calculation is finished. All selection and structural functions can be used on an array containing futures without blocking to wait for the result, as long as no actual data values are required. Meanwhile, other processes (or computers!) could be finishing the evaluation in the background.

It was captivating to observe output being printed and check the status while the calculation work progressed, or accessing parts of an array while other parts still reported as in progress. Morten even had a graphical representation of how load is distributed across processor cores.

Maybe the most exciting thing about all this is that we will soon welcome several new APL squiggles; Isolate (¤) and Parallel (). And at the end of the day, Morten even pulled a pet (pun intended) card out of his sleeve: A Schrödinger’s cat operator! The symbol is of course a question mark in a box (), signifying how the evaluation of the cat’s life (or in our case – the result) does not even begin until actually queried. This is a novel way to specify specific lazy evaluation.

All in all, an exciting look at what possibilities for parallelism the future will bring.

Building a Web Application with MiServer

IMG_4875
One of Sunday’s tracks was Brian Becker’s engaging workshop on MiServer 3.0 – the newest version of Dyalog’s APL-based web application framework that allows you to develop a cross platform application for stand-alone use or delivery via the web.

The motto of MiServer is that everyone who can develop an APL application should also be able to make it available via the web, allowing users to access the application via their favourite browser. In other words, MiServer is browser and platform agnostic.

The main take-away from the morning’s workshop is that if you are used to developing your GUI using ⎕WC, developing the GUI in MiServer 3.0 creates some slightly different rules. First we were showed where to download MiServer, and then Brian guided us though building out very first one-page website. The interactive nature of the workshop captivated us all. Before we knew of it, it was time for lunch.

IMG_4869After lunch – instead of having a sleep inducing talk, Brian challenged us to build a website for a Big Brian’s Burger Bistro offering ordering facility for various products with side orders, and a dashboard to monitor the ordering, product category, speed of order processing, and – most importantly from a management point of view – keep an eye on revenue and which products generate most revenue, with auto update when new orders where processed and paid.

It may have been down to the eating habits of the delegates in terms of their love for burgers, fries etc., but the enthusiasm with which we attacked the challenge was probably more due to how Brian managed to whip up and present MiServer 3.0’s capabilities during the morning. Actually, we got so engrossed in the afternoon that we almost missed our coffee break!

A group of participants was tasked with creating a dashboard, and after some discussion decided on two tables, which would pull data from the business logic, do a bit of calculation, and auto update when new orders got processed.

Hugely entertaining, and we actually managed – courtesy of Rick Proctors swift keyboard handling and deep understanding of APL – to get the code for the dashboard done, working and showcased before we ended the workshop.

We are all looking forward to the MiServer UI Controls presentation on Tuesday morning where we will get a chance to see some the many fancy widgets that MiServer 3.0 offers.

 

Postcard from Dyalog ’15 – Sicily

IMG_4877We’re here, we’re excited and Naxos greeted us with sweltering temperatures and beautiful sunshine on Saturday. Being the oldest Greek settlement in Sicily, founded before Syracuse, around 735 BC, Naxos has a lot to offer. The Atahotel Naxos venue, where the user meeting is taking place, is situated right on the coast with a fantastic view to the impressive Mount Etna – by far the largest of Europe’s active volcanos.

After some people arriving without luggage, due to huge thunderstorms in Rome both Friday and Saturday, we’re now settled and the luggage issues have been sorted. Today, the weather changed to overcast, thunder and light rain and a more bearable temperature. Which was just as well, as Sunday is WORKSHOP day.

IMG_4894You will find more detailed blog posts from a couple of today’s workshops:

Another noticeable workshop today was Dan Baronet’s Recent Language Features in Practice, covering the recently added operators Rank () and Key ().

Tomorrow the user meeting sessions kick off, and we’re anticipating some hugely interesting and engaging days. This year, the Viking challenge has been swapped for an excursion to Mount Etna, which is scheduled for Wednesday afternoon. Sensible walking shoes recommended. We are looking forward to loads a pictures with ‘thumbs up’ when we get to the top!

Postcard from Dyalog ’14 – Thursday

This is the last day of Dyalog ’14 so this is the last of six daily postcards from Eastbourne – we hope you have enjoyed getting a flavour of things as they happened. We expect there will be more discussion to come once we’re back and rested!

Last Night’s Banquet

Yesterday culminated in the traditional banquet dinner – the highlight of the social element of the user meeting. This year it was held at The Grand Hotel Eastbourne and consisted of a Champagne reception and formal three course dinner with live music from a string trio.

During the banquet the winners of the scavenger hunt were presented with their prizes – the ducks and other “British” items we saw on Sunday’s postcard! There was also a quiz to identify Dyalog employees from a list of clues. Exquisite food and wonderful company meant an exceptionally late night (early morning!) …

Delegates arrive at The Grand Hotel

Delegates arrive at The Grand Hotel

Banquet Dinner

Banquet Dinner

Team Herring celebrate their win in the quiz

Team Herring celebrate their win in the quiz

Moris and Luana with their prizes

Moris and Luana with their prizes

Discussion Point: Read-only Component Files

In his presentation today on component files and the Dyalog File Server (DFS), Richard mentioned a perhaps little-known problem with restricting access permissions on component files for users that nominally only ever read them. Setting read-only permission at the file level – that is, imposed by the operating system – could add a level of security to the file but it is in fact the case that component file reads can occasionally require write access to the file.

For example, suppose User A is reading and writing to a journaled component file and User B is simply ⎕FREADing it. If User A is interrupted whilst updating the file (perhaps the APL session is killed or a network connection is lost) and User B is the next to read it then as part of that read the interpreter will detect the journal and complete the unfinished update – a process which requires write access. If the systems administrator who set up the file permissions for User B had only granted read-only file permission to the file then the ⎕FREAD would fail.

Richard explained that this restriction might be lifted in future and that read operations would be truly read only – even with a journal present – but there is another issue to be aware of too. Access to component files is controlled using file locks. As might be expected, exclusive access is obtained on an exclusive tie and shared access is obtained on a share tie. In addition, writes to a share tied file require exclusive access for the duration of the write so that no other client is able to read it while it is being updated. Exclusive locks are write locks and thus require write permission. Therefore, if there is no write access to the file the interpreter cannot exclusively tie it – it can only share tie it.

Where file access has to be locked down as securely as possible, the DFS is able to resolve the above issues and more besides – client access to all component files is only possible via APL, users are fully authenticated and individual rights are controlled by the access matrix.

And Also …

Some of the other things we saw and heard today:

  • Andy showed us how to close trace windows without cutting back the stack
  • Ziggi pointed out that a loss rate of one in 700 is acceptable when delivering wine but not when delivering children
  • We would like your suggestions and comments about the user meeting programme and structure, and preferred location for next year – please email us at conference@dyalog.com

Eastbourne

Eastbourne has not let us down and even the British weather (which is always unpredictable) defied the odds and rendered the ponchos and umbrellas unnecessary. The View Hotel started life as a conference centre and was purpose built for holding meetings such as ours. We have certainly found it to be an ideal venue and feedback from delegates has been overwhelmingly positive.

The View Hotel

The View Hotel

The View Hotel

The View Hotel

The picture of the pier below was carefully framed not to show the fire damaged part! The fire that destroyed part of it back in July was headline news; the pier is closed as a result but should partially reopen next week. Perhaps we will have to return when it is fully restored!

Eastbourne Pier

Eastbourne Pier

Bandstand with The View Hotel behind

Bandstand with The View Hotel behind

Signing Off

And so Dyalog ’14 has come to an end. It has been the best-attended Dyalog conference anyone can remember and reflects the growing success of Dyalog and its users. Here begins the countdown to Dyalog ’15!

Postcard from Dyalog ’14 – Wednesday

It’s been another busy day at Dyalog ’14; there’s lots to write about and little time to do it!

Winners of the International APL Programming Contest 2014

This year, 710 registrations were received for the annual APL Programming Contest and the two main winners – Emil Bremer Orloff, student entrant from Aarhus Universitet, and Iryna Pashenkovska, professional entrant from SimCorp and a recent Engineering Mathematics graduate of National Taras Shevchenko University of Kyiv – won invitations to Dyalog ’14 as part of their prizes where they each made presentations about their contest submissions. Look forward to the videos of their presentations which will be available shortly!

Emil Bremer Orloff

Emil Bremer Orloff

Iryna Pashenkovska

Iryna Pashenkovska

Emil Bremer Orloff

Emil Bremer Orloff

Iryna Pashenkovska

Iryna Pashenkovska

Discussion Point: Data Binding – Reloaded

John Daintree showed more aspects of Dyalog’s data binding. Did you know that you can bind to XML stored in files? Did you know that you can create DataTables to bind to an APL matrix? The cddb database sample was re-worked to allow the GUI to bind to the data in a matrix as an alternative to vectors of namespace. Syncfusion controls were used to visually explore the cddb data.

John discussed changes that might come in 14.1, including better support for bound matrices.

Discussion Point: Syncfusion Controls

Late yesterday, Brian Becker showed how to take advantage of the Syncfusion Javascript controls that are bundled with Dyalog v14.0 on all platforms. This morning, Morten Kromberg and Michael Hughes followed up with demonstrations of how easy it is to enhance applications with Syncfusion controls for Windows Presentation Foundation, which are included with the Windows editions of version 14.0.

With Dyalog (as in any Microsoft.NET language), you have the choice of instantiating and manipulating WPF objects with APL language statements, or using declarative eXtensible Application Markup language, which can either be edited by UI design experts using WYSIWYG tools, or (of course) generated from APL.

A Syncfusion RangeSliderControl

A Syncfusion RangeSliderControl

A Syncfusion “Sparkline”

A Syncfusion “Sparkline”

Code sample for the RangeSlider:


     ∇ window←RangeSliderControl values;⎕USING;r;t
[1]   ⍝ Simple Sparkline example - inspired by Dick Bowman
[2]   ⍝ More at http://www.apldapldoo.info/apl/APL/Tutorials/WPF/SyncFusion/overview.html
[3]   ⍝ values: Numeric vector of values
[4]
[5]    :Section xaml
[6]        xaml←ScriptFollows ⍝ Picks up following comments as data
[7]    ⍝<Window
[8]    ⍝  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
[9]    ⍝  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
[10]   ⍝  xmlns:sftools="clr-namespace:Syncfusion.Windows.Tools.Controls;assembly=SyncFusion.Tools.WPF"
[11]   ⍝  Title="Syncfusion Range Slider" SizeToContent="Height" Width="400">
[12]   ⍝  <StackPanel>
[13]   ⍝    <sftools:RangeSliderControl Name="rangeslider" />
[14]   ⍝    <TextBox Name="textbox" MinHeight="25" MinWidth="100"
[15]   ⍝             HorizontalContentAlignment="Right"
[16]   ⍝             VerticalContentAlignment="Bottom"
[17]   ⍝             Background="Turquoise"/>
[18]   ⍝  </StackPanel>
[19]   ⍝</Window>
[20]   :EndSection
[21]
[22]   window←LoadXAML xaml
[23]   (r t)←window.FindName∘⊂¨'rangeslider' 'textbox'
[24]   ⎕USING←,⊂'Syncfusion.Windows.Tools.Controls,Syncfusion/4.5/Syncfusion.tools.WPF.dll'
[25]   r.onRangeChanged←'RangeChanged'
[26]   r.Range←⎕NEW DoubleRange values
[27]   window.Show
[28]
[29]   window.(slider textbox range)←r t r.Range ⍝ expose extra "properties"
     ∇

James Bond or Johnny English?

This year we enrolled our delegates in ‘Spy School’ and gave them the chance to be a spy for a few hours. In an interactive mission that included GPS tracking, teams were given a tablet preloaded with a series of spy related challenges that took place around the local area. The challenges were only activated when the GPS verified the team was in exactly the right location and points were awarded for completing each challenge successfully – the team with most points was the winner. To pile on the pressure, each team had an interactive map showing not only their location but also that of the other teams along with a live scoreboard.

The challenge certainly brought out the competitive side of the delegates and Eastbourne is probably still recovering from the shock of ten teams of APLers carrying out their covert missions around the town centre and promenade. When the final scores were in, it was a very closely run thing with Team Sprat in second place and Team Dover Sole, headed up by a super competitive John Daintree, as the eventual winners. At the final de-briefing we got to see the videos and photos submitted by each team which showed great creativity along with some very over active imaginations! Everyone had a fantastic afternoon with plenty of laughter and not too many injuries (although I am led to believe there may be a broken fingernail or two) and a big thank you to Pretty Clever Events for running the show.

Team Plaice

Team Plaice

Team Dover Sole

Team Dover Sole

Team Dover Sole take a selfie at the  award presentation

Team Dover Sole take a selfie at the award presentation

Coming up…

Today is due to end with the banquet dinner in the Grand Hotel which we’ll try to include in tomorrow’s postcard.

We conclude events tomorrow with two Dyalog and two user presentations in the morning followed by the closing session at which all of Team Dyalog will be present and open to questions. In the afternoon are the remaining Technical Workshops.

Postcard from Dyalog ’14 – Tuesday

The weather predictions have been spot on so far; it’s been another lovely sunny day at Eastbourne – here was the scene this morning at sunrise:

Sunrise in Eastbourne

Sunrise in Eastbourne

It was another busy day of presentations – read on for some highlights! After dinner tonight the evening will be rounded off with Morten and his Dancing ‘Bots – controlling two robots in parallel.

Discussion Point: Chart Design

As he demonstrated Chart Wizard – a front-end to help building SharpPlot charts – Nic pointed out a few useful tips about chart design:

Gitte Christensen and Nic Delcros

Gitte Christensen and Nic Delcros

  • Transparency is your best friend
    SharpPlot supports transparency, which not only allows multiple layers of drawings to be readable, but also lightens the global visual impression, making charts more 21st-century
  • 3D is not as cool as it looks
    The 2D alternatives (Bubble chart, Contour plot) are generally more readable, because perspective can hide data in automated 3D charts.
  • Logarithmic scales are too often neglected
    They can improve readability in many cases, for example stock value, performance comparisons, or wealth repartition
  • Box & Whiskers for categorised data exploration
    A very effective tool to explore the repartition of values amongst a un-modelled, yet categorised, numerical dataset.

Paul’s Poncho

MD of Optima Systems models a rain poncho

MD of Optima Systems models a rain poncho

The registration packs that delegates received included a stick of rock and a rain poncho generously donated by Optima Systems. As these items were unfamiliar to many from overseas, Paul Grosvenor – Optima Systems’ Managing Director – modelled them for us.

Behind The Scenes

Jason in the AV room

Jason in the AV room

Behind each presentation at Dyalog ’14 is a team of people taking care – hopefully mostly unnoticed – of all the AV requirements. Jason and Jonathan are working full-time in the AV suite and they are being assisted by a rota of Dyalog staff and Liam Flanagan, who has temporarily returned to us for the event, as microphone runners and camera operators. In total there are ten audio feeds and three video feeds, and every presentation is being recorded.

Generally, each presenter has different software and hardware needs and we allow everyone to use their own kit. In past years this has meant that changeovers have sometimes been less than seamless so this time we are experimenting with a system that allows one presenter’s equipment to be set up while the previous session is still taking place – this is giving a really noticeable speed-up in the changeover times.

Over the coming weeks the recordings will be edited and released online. The presentations will be released at the rate of approximately two per week as each generally takes around six to eight hours to edit and needs to be reviewed and approved by the presenter.

And Also …

Some of the many other things we saw and heard today:

  • Kai Jaeger told us that software support such as Acre is essential to prevent chaos when there are multiple programmers working on the same project
  • Gianfranco Alongi observed that maintenance is not cool, but said that if your product is not being maintained then the chances are it is already dead
  • SimCorp maintains 1.7 million lines of APL code
  • Stephen Mansour said that the biggest difference between Dyalog and other stats tools such as R is the existence of defined operators
  • Nick did not use Powerpoint in his presentation but instead had a text-only equivalent that he wrote himself in coffeescript
  • Aaron Hsu announced that his Co-dfns compiler would soon be available on GitHub for download
  • Brian demonstrated MiServer 3’s ability to display CIA data
  • Bjørn introduced the Dyalog Cryptographic Library which is now available for use
Lunch break

Lunch break

Did You Know?

John Scholes and Geoff Streeter started writing the Dyalog APL interpreter this day in 1981?

Tomorrow…

Tomorrow morning’s schedule includes two Dyalog and two user presentations. The afternoon will be dominated by the Viking Challenge (don’t forget to wear outdoor clothes and shoes that are suitable for walking!) followed by the prize ceremony for the International APL Programming Contest 2014. The two winners (one student and, for the first time, one non-student) will talk about how they achieved their victory. Everyone is encouraged to give this next generation of APLers their full support. The banquet will round off the day.