Is it Sorted?

Motivation

I have been working on the Dyalog APL quicksort implementation. The following programming puzzle arose in the process of doing the QA for this work.

`⍵` is a simple array. Write a function `sorted`, without using `⍋` or `⍒`, such that `sorted ⍵` is 1 if `⍵` is sorted in ascending order and 0 otherwise.

The point about not using grade is that this is supposed to be an independent check that grade is correct (remains correct) after the new work.

Real Vectors

The simplest case is when `⍵` is a numeric vector. If furthermore `⍵` are not complex numbers (a case addressed later), then

``````   ∧/ 2≤/⍵
``````

each item being less than or equal to the next one, checks that `⍵` is sorted. Since `⍋` uses exact comparisons, here we must set `⎕ct←⎕dct←0`. Morever, in order that decimal floating-point numbers (DECFs) be compared correctly, here `⎕fr←1287`.

Real Arrays

More generally, when `⍵` is a non-complex numeric matrix, we must check that each row precedes or is equal to the next row. If `c` and `d` are consecutive rows, then corresponding items are compared and at the first item where they differ, `c[i]` must be less than `d[i]`.

``````   ~ 0 ∊ (2>⌿⍪⍵) ⍲ <\ 2≠⌿⍪⍵
``````

The expression incorporates two refinements:

• If `⍵` is not a matrix, first apply `⍪⍵`.
• Instead of checking `c[i]` is less than `d[i]`, check that `c[i]` is not greater than `d[i]`. This finesses the case where `c≡d` and there is no first item where they differ; that is, the case where `<\2≠⌿⍪⍵` is all 0s for that row.

`<\`on a boolean vector has 0s after the first 1, (and is all 0 if there are no 1s). Therefore, `<\2≠⌿⍪⍵` finds the first item (if any) where one cell differs from the next cell, and that item must not be greater than the corresponding item in the next cell.

For example:

``````   x←?97 3⍴10

{~ 0 ∊ (2>⌿⍪⍵) ⍲ <\ 2≠⌿⍪⍵} x
0
{~ 0 ∊ (2>⌿⍪⍵) ⍲ <\ 2≠⌿⍪⍵} x[⍋x;]
1
``````

(Muse: since `x` above are random numbers, there is a possibility that it is sorted and the first test above can be 1. But: if each elementary particle in the visible universe were a computer and every nanosecond each of them creates a random matrix and tests it for sortedness as above, running from the beginning of the time to the end of our lives, it is still a very safe bet that no 1 would result.)

For integer arrays, there is an alternative of using the signs of the arithmetic difference between successive cells:

``````   {~ 0 ∊ 1≠t×<\0≠t← × 2-⌿⍪⍵} x[⍋x;]
1
``````

(The sign where consecutive cells first differ must not be 1.) However, computing the difference on floating point numbers can founder on overflow:

``````   ⊢ x←¯1 1×⌊/⍬
¯1.79769E308 1.79769E308

{~ 0 ∊ 1≠t×<\0≠t← × 2-⌿⍪⍵} x
DOMAIN ERROR
{~0∊1≠t×<\0≠t←×2-⌿⍪⍵}x
∧
``````

Complex Numbers

Two complex numbers are ordered first by the real parts and then by the imaginary parts. (This is part of the TAO extension implemented in Dyalog APL version 17.0.) Therefore, a complex array can be tested for sortedness by testing an equivalent real array with each number replaced by their real and imaginary parts, thus:

``````   (¯1⌽⍳1+⍴⍴⍵) ⍉ 9 11∘.○⍵
↑9 11∘○¨⍵
9 11○⍤1 0⊢⍵
``````

Although the second expression is the shortest, it is less efficient in time, space, and number of `getspace` calls. The last expression is favored for its brevity and performance.

The number of `getspace` is a worthwhile measure. Part of the QA process is a rather stringent procedure called the “Shuffle QA”. The entire Shuffle QA takes several weeks to run and its running time is directly related to the number of `getspace`.

Character Arrays

None of the functions `< ≤ ≥ > - ×` are permitted on characters. This is solved by application of `⎕ucs`, converting characters to integers while preserving the ordering.

Putting It All Together

``````sorted←{
⎕ct←⎕dct←0
⎕fr←1287
d←10|⎕dr ⍵
d∊0 2: ∇ ⎕ucs ⍵
d=9:   ∇ 9 11○⍤1 0⊢⍵
~ 0 ∊ (2>⌿⍪⍵) ⍲ <\ 2≠⌿⍪⍵
}
``````

Other Considerations

That `⍵⌷⍨⊂⍋⍵` is sorted is a necessary but not sufficient condition that `⍋⍵` is correct. For example, an “adversary” can supply the following results for `⍋⍵` so that `⍵⌷⍨⊂⍋⍵` is sorted:

``````?≢⍵
(≢⍵)⍴?≢⍵
¯1↓⍋⍵
∊ i {⊂⍵[?⍨≢⍵]}⌸⍨ ⍵⌷⍨⊂i←⍋⍵
``````

The last expression randomly permutes the grade indices of equal cells, a result which violates the requirement that grade indices of equal cells are in ascending order. That is, grade must be stable.

In Dyalog APL version 17.0, grade has been extended to work on non-simple arrays, the much discussed TAO, total array ordering. Checking that a non-simple array is sorted without using grade requires facilities discussed in the paper TAO Axioms and is beyond the scope of this note.

`⎕io=0` is assumed throughout. The essay talks only about `⍋` but the same ideas apply to `⍒`.

Background

`⍋` has the distinction of being the first (in 1980) APL primitive function defined on major cells: the result orders items of a vector, rows of a matrix, planes of a 3-d array, etc. In the ordering major cells are compared in ravelled order, with leading items being more significant than trailing (lexicographic ordering). Moreover, in dyadic grade `⍺⍋⍵`, `⍺` specifies “alphabets” to be used in comparing the items of character array `⍵`.

Dyadic grade has always been an APL primitive which is hard for me to understand, in that way kind of like dyadic transpose ☺. I sat down to really understand it, starting from the simplest cases to the general case. The following is a record of my explorations.

Vector Left Argument

``````   gv← {⍋⍺⍳⍵}

a0← 'abcdefghij'
x0← 'chthonic'

a0 gv x0
0 7 1 3 6 2 4 5
a0 ⍋ x0
0 7 1 3 6 2 4 5

x0 ⌷⍨ ⊂ a0 gv x0
cchhiton
``````

That is, grade the indices of `⍵` in `⍺`. If an item of `⍵` is not in `⍺` then its index is `≢⍺`.

Higher-Rank Left Argument with Unique Items

The coordinates of `A[i;j;k;…]` or `A[⊂i,j,k,…]` is the vector `i,j,k,…`. The phrase `⍳⍴A` produces the array of coordinates. For example, if `⍺` is the `(2 26)`-matrix of the upper and lower case English letters,

``````   ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
``````

the corresponding coordinates are

``````   ┌───┬───┬───┬───┬───┬───┬───┬───┬───┬   ┬────┬────┐
│0 0│0 1│0 2│0 3│0 4│0 5│0 6│0 7│0 8│   │0 24│0 25│
├───┼───┼───┼───┼───┼───┼───┼───┼───┼ … ├────┼────┤
│1 0│1 1│1 2│1 3│1 4│1 5│1 6│1 7│1 8│   │1 24│1 25│
└───┴───┴───┴───┴───┴───┴───┴───┴───┴   ┴────┴────┘
``````

If the items of `⍺` are unique,

``````   gu← {⍋ 0 2 1 ⍉ (⊂(,⍺)⍳⍪⍵) ⌷ ⌽ (⍴⍺) ⍪⍨ ⍉(⍴⍺)⊤⍳×/⍴⍺}
``````

That is, `⍺⍋⍵` obtains as the grade of the reversed coordinates of `⍵` in `⍺`. (If an item does not occur in `⍺`, its coordinates are `⍴⍺`.) The `⌽` implements that in `⍺`, the first axis is least significant and the last axis is most significant. For the `(2 26)`-matrix above, case (the first axis) is less significant than `A-Z` and `a-z` (the last axis).

``````   ⊢ a1←' ',⎕av[(⎕av⍳'Aa')∘.+⍳26]
ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz

Jay
roger
Roger
jay

a1 gu x1
4 3 0 5 2 1
a1 ⍋ x1
4 3 0 5 2 1

x1 ⌷⍨ ⊂ a1 gu x1
Jay
jay
Roger
roger
``````

Higher-Rank Left Arguments

Suppose `⍺` does have duplicates? For purposes of `⍋`, the coordinates of an item `c` are

``   ⌊⌿(c=,⍺)⌿↑,⍳⍴⍺``

That is, the minimum of coordinates of all items equal to `c`. Note that the expression also works if `c` is a unique item. Therefore, for a general `⍺`, with or without duplicates, `⍺⍋⍵` obtains as

``````   gr← {⍋ 0 2 1 ⍉ (⊂(∪,⍺)⍳⍪⍵) ⌷ ⌽ (⍴⍺) ⍪⍨ (,⍺) {⌊⌿⍵}⌸ ⍉(⍴⍺)⊤⍳×/⍴⍺}
``````

The “minimum of coordinates” computation is exploited to effect equal coodinates for disparate characters. For example, an ordering where upper and lower case are significant but diacritical marks are not, can be implemented as follows:

``````   A    ⍝ A has a leading blank column
AÀÁÂÃÄÅBCÇDEÈÉÊËFGHIÌÍÎÏJKLMNÑOÒÓÔÕÖØPQRSTUÙÚÛÜVWXYÝZ
aàáâãäåbcçdeèéêëfghiìíîïjklmnñoòóôõöøpqrstuùúûüvwxyýz
À       Ç  È       Ì        Ñ Ò                   Ý
Á       ç  É       Í        ñ Ó                   ý
Â          Ê       Î          Ô
Ã          Ë       Ï          Ö
Ä          è       ì          Õ
Å          é       í          Ø
à          ê       î          ò
á          ë       ï          ó
â                             ô
ã                             õ
ä                             ö
å                             ø
⍴A
14 54

('È'=,A)⌿↑,⍳⍴A                ('è'=,A)⌿↑,⍳⍴A
0 13                          1 13
2 12                          6 12
⌊⌿('È'=,A)⌿↑,⍳⍴A              ⌊⌿('è'=,A)⌿↑,⍳⍴A
0 12                          1 12

('E'=,A)⌿↑,⍳⍴A                ('e'=,A)⌿↑,⍳⍴A
0 12                          1 12
⌊⌿('E'=,A)⌿↑,⍳⍴A              ⌊⌿('e'=,A)⌿↑,⍳⍴A
0 12                          1 12
``````

`'È'` occurs twice in `A` with coordinates `0 13` and `2 12`. The coordinates assigned to `'È'` are the minimum of these, `0 12`. In contrast, `'E'` occurs once and its coordinates are `0 12`, the same as those for `'È'`. Therefore, `'E'` and `'È'` are considered equal for purposes of dyadic grade. Similarly, `'e'` and `'è'` have coordinates `1 12` and are considered equal by `⍋`, but they follow `'E'` and `'È'` (because their coordinates are `0 12`).

For example:

``````   ⊢ x← ↑' '(≠⊆⊢)' roger adàm Röger rÖger Adåm JÃY JAY JÃY adåm adàm'
roger
Röger
rÖger
JÃY
JAY
JÃY

A gr x
4 1 8 9 5 6 7 2 3 0
A ⍋ x
4 1 8 9 5 6 7 2 3 0

x ⌷⍨⊂ A gr x
JÃY
JAY
JÃY
Röger
rÖger
roger
``````

Lest you think that dyadic grade in its full generality suffices to implement any ordering: in “telephone book” ordering, “1600 Pennsylvania Avenue” and “Unter den Linden 23” are ordered as if 1600 were spelled out as “Sixteen Hundred” and 23 as “Dreiundzwanzig”. A program to do that ought to be très amusant.

Code Archeology

The above code are improved versions of what appeared in Peter Wooster, Extended Upgrade and Downgrade, SHARP APL Technical Notes 35, I.P. Sharp Associates, 1980-09-15. It is interesting to study the code from the two eras. (The code from 1980 is lightly edited for executability and clarity.)

2018

``````gv← {⍋⍺⍳⍵}
gu← {⍋ 0 2 1 ⍉ (⊂(,⍺)⍳⍪⍵) ⌷ ⌽ (⍴⍺) ⍪⍨ ⍉(⍴⍺)⊤⍳×/⍴⍺}
gr← {⍋ 0 2 1 ⍉ (⊂(∪,⍺)⍳⍪⍵) ⌷ ⌽ (⍴⍺) ⍪⍨ (,⍺) {⌊⌿⍵}⌸ ⍉(⍴⍺)⊤⍳×/⍴⍺}
``````

1980

``````eu← {d⊤⍳×/d←⍴⍵}
er← {¯1+÷(÷1+d⊤⍳×/d←⍴⍵)⌈.×a∘.=a←,⍵}

fv← {⍋⍺⍳⍵}
fu← {⍋(⍒0 1,1↓0×⍳⍴⍴⍵)⍉(⊖(eu ⍺),⍴⍺)[;(,⍺)⍳⍵]}
fr← {⍋(⍒0 1,1↓0×⍳⍴⍴⍵)⍉(⊖(er ⍺),⍴⍺)[;(,⍺)⍳⍵]}
``````
 `gv, fv` vector left argument `gu, fu` higher-ranked left argument with unique items `gr, fr` higher-ranked left argument

In the sequence `gv gu gr`, a function is more general than the preceding one and subsumes it. Likewise `fv fu fr`.

Comparison of the code illustrates advances in APL between 1980 and 2018:

• `{⌊⌿⍵}⌸ `minimum of major cells corresponding to identical keys
• `∪      `unique items
• `⍪⍵     `ravel major cells
• `⍺⍪⍵    `catenate on first axis
• `⍨      `commute operator
• dfns

Alternatives

If a left argument is large and complicated and is used repeatedly, it may be worthwhile for the APL interpreter to perform precomputations on it. Thus:

``````   U← ∪,A
C← ⌽ (⍴A) ⍪⍨ (,A) {⌊⌿⍵}⌸ ⍉(⍴A)⊤⍳×/⍴A

⍴U        ⍴C
107       108 2

⍪U        C
0  0
A          1  0
À          1  0
Á          1  0
Â          1  0
Ã          1  0
Ä          1  0
Å          1  0
B          8  0
C          9  0
Ç          9  0
…           …
x         50  1
y         51  1
ý         51  1
z         53  1
14 54

gp← (U C)∘{U C←⍺ ⋄ ⍋0 2 1⍉C[U⍳⍪⍵;]}

gp x
4 1 8 9 5 6 7 2 3 0
A ⍋ x
4 1 8 9 5 6 7 2 3 0
``````

It makes sense that the number of columns in the coordinate matrix `C` is equal to the rank of the alphabet array `A`: The rank is the number of dimensions, a-z, upper/lower case, color, etc.; each row of `C` is a vector of the numeric value for each dimension.

With 20/20 hindsight, the above code can be seen as an argument against defining dyadic grade to do ordering with specified alphabets. After all,

``````   ⍺⍋⍵  ←→  ⍋0 2 1⍉C[U⍳⍪⍵;]
``````

and specifying `U` and `C` directly makes the computation easier to understand, easier to use, and as it happens is faster than the primitive in the current implementation. The inverse calculation, from `U C` to the alphabet array `A`, is an amusing bit of code left as an exercise for the reader☺.

One can further argue that the current definition of dyadic grade precludes an alternative attractive but incompatible definition:

``````   ⍺⍋⍵  ←→  ⍺⌷⍨⊂⍋⍵
``````

That is, order `⍺` by the grade of `⍵`, whence `⍋⍨` sorts. In Dyalog APL version 17.0, monadic grade is extended to work with a TAO (total array ordering). With a TAO and this alternative definition, `⍋⍨` sorts any array.

The present exposition exposes a difficulty with extending the current dyadic grade to work with TAO: It is axiomatic that monadic grade compares cells itemwise, stopping at the first pair of unequal items. Dyadic grade does not do that in general. For example, with an upper and lower case alphabet, you don’t stop comparing `'Rogerz'` and `'rogers'` on encountering `'R'` and `'r'`.

Linear Interpolation

`⎕io=0` assumed throughout; works in 1-origin with the obvious modifications.

Introduction

On Wednesday, a question arrived via Dyalog Support from an intern in Africa: If `M` is the matrix on the left, use linear interpolation to compute the result on the right.

``````   1 20         1 20
4 80         2 40
6 82         3 60
4 80
5 81
6 82
``````

Linear Interpolation

Two points `(x0,y0)` and `(x1,y1)` specify a line; for any `x` there is a unique `y` on that line (assuming `x0≠x1`). The equation for the line derives as follows, starting from its slope `m`:

``````   m = (y1-y0) ÷ (x1-x0)
(y-y0) = m × (x-x0)
y = y0 + m × (x-x0)
``````

Therefore, if `⍺` is a 2-by-2 matrix of the two points and `⍵` are the x-values to be interpolated, then:

``````   g ← {(⊃⌽⍺)+(⍵-⊃⍺)÷÷/-⌿⍺}

⊢ M←1 4 6,⍪20 80 82
1 20
4 80
6 82

M[0 1;] g 2 3
40 60
M[1 2;] g 5
81
``````

A New Twist, A New Solution

The problem as posed implicitly required that:

• The x-values are the positive integers bounded by `⊃⊖M`.
• Appropriate rows of the matrix are selected for a given x-value.
• The missing x-values and their interpolations are “slotted back” into the argument matrix.

These requirements are best met by `⍸`, interval index, a relatively new primitive function introduced in Dyalog APL version 16.0. The left argument `⍺` must be sorted and partitions the universe into disjoint contiguous intervals; `⍺⍸⍵` finds the index of the interval which contains an item of `⍵`. The result is `⎕io` dependent.

For the given matrix `M`, the partition (of the real numbers in this case) is depicted below. As in conventional mathematical notation, `[` denotes that the interval includes the left end-point and `)` denotes that the interval excludes the right end-point.

``````          1        4      6
─────────)[───────)[─────)[──────────
¯1       0       1       2

v←¯5 0 1 2.5 6 3 4 5 9 8 7

1 4 6 ⍸ v
¯1 ¯1 0 0 2 0 1 1 2 2 2

v ,[¯0.5] 1 4 6 ⍸ v
¯5  0 1 2.5 6 3 4 5 9 8 7
¯1 ¯1 0 0   2 0 1 1 2 2 2
``````

With `⍸` in hand, the problem can be solved as follows:

``````interpol←{
(x y)←↓⍉⍵
m←m,⊃⌽m←(2-/y)÷(2-/x)
j←0⌈x⍸i←1+⍳⊃⌽x
i,⍪y[j]+m[j]×i-x[j]
}

interpol M
1 20
2 40
3 60
4 80
5 81
6 82
``````

The problem of x-values less than the first end-point is finessed by applying `0⌈` to the interval indices, and that of x-values greater than or equal to the last end-point is finessed by repeating the last slope `m←m,⊃⌽m`.

It is possible to do the interpolation only on the missing indices (2 3 5 in this case) and insert them into the argument matrix. It seems neater to simply interpolate everything, and in so doing provide a check that the interpolated values equal the values given in the argument.

An Alternative Interpolation

Interpolating according to two selected rows of a matrix of points treats the function as piecewise linear, with sharp inflection points where the lines join (different slopes between adjacent lines). A “holistic” alternative approach is possible: the matrix can be interpreted as specifying a single line and the interpolation is according to this single line. The `⌹` primitive function computes the coefficients of the line which best fits the points:

``````   ⎕rl←7*5  ⍝ for reproducible random numbers

⊢ M←t,⍪(?7⍴5)+¯17+3×t←?7⍴100
35  89
98 278
19  44
4  ¯5
62 170
49 133
25  59

M[;1] ⌹ 1,M[;,0]    ⍝ y-intercept and slope
¯15.3164 2.99731

interpola ← {(1,⍤0⊢⍵)+.×⍺[;1]⌹1,⍺[;,0]}

M[;1] ,[¯0.5] M interpola M[;0]
89      278    44      ¯5       170     133     59
89.5895 278.42 41.6325 ¯3.32713 170.517 131.552 59.6164

M interpola 33 35 37 39.7
83.5949 89.5895 95.5841 103.677
``````

Finally

Our best wishes to the intern. Welcome to APL!

Permuting Internal Letters

Friday Afternoon

It’s something of a custom in Dyalog to send a “fun” e-mail to the group on Friday afternoons. My gambit for this past Friday was:

``````   x ←' according to research it doesn''t matter'
x,←' what order the letters in a word are'
x,←' the human mind can still read it'
x,←' the only important thing is that'
x,←' the first and the last letters are in the right place'

∊ ' ',¨ {(1↑⍵),({⍵[?⍨≢⍵]}1↓¯1↓⍵),(-1<≢⍵)↑⍵}¨ (' '∘≠ ⊆ ⊢) x
aroidnccg to reasrech it dneso't mttear waht order the ltreets
in a wrod are the hmaun mnid can sltil read it the olny
imartpnot tihng is that the fsrit and the lsat lterets
are in the rhigt palce``````

Code Golfing

The code golfers were not long in responding:

``````   ∊{' ',⍵[1,(1+?⍨0⌈¯2+n),n/⍨1<n←≢⍵]}¨(' '∘≠⊆⊢)x     ⍝ fa
∊{' ',⍵[∪1,{⍵,⍨?⍨⍵-1}≢⍵]}¨(' '∘≠⊆⊢)x              ⍝ fb
'\S+'⎕r{{⍵[∪1,{⍵,⍨?⍨⍵-1}≢⍵]}⍵.Match}x             ⍝ fc
∊{' ',⍵[n↑1,(1+?⍨0⌈¯2+n),n←≢⍵]}¨(' '∘≠⊆⊢)x        ⍝ fd
``````

In defense of the original expression, it can be said that it follows the pattern:

• cut into words `           (' '∘≠ ⊆ ⊢) x ` also` ' '(≠⊆⊢)x`
• permute each word`      {(1↑⍵),({⍵[?⍨≢⍵]}1↓¯1↓⍵),(-1<≢⍵)↑⍵}¨`
• undo the cut`             ∊ ' ',¨`

That is, the pattern is “permute under cut”. Moving the `' ',` into the permuting function, while shortening the overall expression, obscures this pattern.

Golfing for Speed

One of the code golfers was undaunted by the highfalutin’ functional programming argument. Changing tack, he claimed to be golfing for speed (while himself travelling at 900 kph):

``````fe←{
p←1+0,⍸⍵=' '           ⍝ start of each word
n←0⌈¯3+¯2-/p,2+≢⍵      ⍝ sizes of windows to be shuffled
⍵[∊p+?⍨¨n]@((⍳¯1+≢⍵)~p∘.-0 1 2)⊢⍵
}

cmpx 'fe x' 'fb x' 'fc x' 'fd x'
fe x → 4.27E¯5 |    0% ⎕⎕⎕⎕
* fb x → 1.05E¯4 | +145% ⎕⎕⎕⎕⎕⎕⎕⎕⎕
* fc x → 3.39E¯4 | +692% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
* fd x → 1.11E¯4 | +159% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
``````

The question was then posed whether the algorithm can be “flattened” further; that is, whether the expression `∊p+?⍨¨n` in function `fe` can be done without each. The answer is of course yes (else I wouldn’t be writing this blog post :-). Flattening, or avoiding the creation of nested arrays, has the potential to reduce memory consumption and increase efficiency, because there is more potential for the interpreter to perform optimized sequential or even parallel operations.

Partitioned Random Permutations

In the old days, before general arrays, before the each and rank operators, ingenious techniques were devised for working with “partitioned” arrays: A boolean vector with a leading 1 specifies a partition on a corresponding array with the same length, basically what we can now do with `⊂` or similar facility. A detailed description can be found in Bob Smith’s APL79 paper A Programming Technique for Non-Rectangular Data.

``````   p←1 0 0 1 1 0 0 0 0 0
v←3 1 4 1 5 9 2 6 53 58

p⊂v
┌─────┬─┬─────────────┐
│3 1 4│1│5 9 2 6 53 58│
└─────┴─┴─────────────┘

+/¨ p⊂v
8 1 133
t-¯1↓0,t←(1⌽p)/+\v
8 1 133

∊ +\¨ p⊂v
3 4 8 1 5 14 16 22 75 133
s-(t-¯1↓0,t←(1⌽p)/⍳⍴p)/¯1↓0,(1⌽p)/s←+\v
3 4 8 1 5 14 16 22 75 133
``````

The last two sets of expressions illustrate how “partitioned plus reduce” and “partitioned plus scan” can be computed, without use of general arrays and without each.

At issue is how to do “partitioned random permute”. Answer: `⍋(n?n)+n×+\p ⊣ n←≢p`.

``````p                1  0  0    1    1  0  0  0  0  0
n?n              5  4 10    6    2  9  8  1  3  7
n×+\p           10 10 10   20   30 30 30 30 30 30
(n?n)+n×+\p     15 14 20   26   32 39 38 31 33 37
⍋(n?n)+n×+\p     2  1  3    4    8  5  9 10  7  6``````

Therefore:

``````ff←{
b←~(⊢ ∨ 1∘⌽ ∨ ¯1∘⌽)' '=⍵  ⍝ internal letters
n←≢p←b/b>¯1⌽b             ⍝ partition vector for groups of same
(b/⍵)[⍋(n?n)+n×+\p]@{b}⍵
}

cmpx 'ff x' 'fe x'
ff x → 1.46E¯5 |    0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
* fe x → 4.19E¯5 | +187% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

x6←1e6⍴x

cmpx 'ff x6' 'fe x6'
ff x6 → 5.16E¯2 |    0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕
* fe x6 → 1.77E¯1 | +243% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
``````

There is a puzzle from 1980 which similarly involved randomly permuting items within groups. See A History of APL in 50 Functions, Chapter 47, Pyramigram. The puzzle solution is amusing also for that fact that it used `-\`.

Afterword

Plobssiy, a txet wshoe words are ilivuddinaly ptemrued is rdeaable mnaliy bseacue in odirrany txet many wrods, eelpclsaiy iptnroamt wrods, are sorht. A curops wtih long and ufnmiailar wdors wuold lkiely be hard to raed atfer scuh pittarmoeun. For eapxmle:

``````   ff t
eyonelsmeary dpihnoeissopt siiuqpeedlasan pniormoasaac oopimoentaoa
ff t
ff t
erloymneseay dhsiepopiosnt seildspqiauean praisoaaonmc oomatneiopoa
``````

Stencil Lives

`⎕io←0` throughout. `⎕io` delenda est.

Stencil

A stencil operator `⌺` is available with Dyalog version 16.0. In brief, stencil is a dyadic operator `f⌺s` which applies `f` to (possibly overlapping) rectangles. The size of the rectangle and its movement are controlled by `s`. For example, enclosing 3-by-3 rectangles with default movements of 1:

``````   ⊢ a←4 5⍴⎕a
ABCDE
FGHIJ
KLMNO
PQRST
{⊂⍵}⌺3 3 ⊢a
┌───┬───┬───┬───┬───┐
│   │   │   │   │   │
│ AB│ABC│BCD│CDE│DE │
│ FG│FGH│GHI│HIJ│IJ │
├───┼───┼───┼───┼───┤
│ AB│ABC│BCD│CDE│DE │
│ FG│FGH│GHI│HIJ│IJ │
│ KL│KLM│LMN│MNO│NO │
├───┼───┼───┼───┼───┤
│ FG│FGH│GHI│HIJ│IJ │
│ KL│KLM│LMN│MNO│NO │
│ PQ│PQR│QRS│RST│ST │
├───┼───┼───┼───┼───┤
│ KL│KLM│LMN│MNO│NO │
│ PQ│PQR│QRS│RST│ST │
│   │   │   │   │   │
└───┴───┴───┴───┴───┘
``````

Stencil is also known as stencil code, tile, tessellation, and cut. It has applications in artificial neural networks, computational fluid dynamics, cellular automata, etc., and of course in Conway’s Game of Life.

The Rules of Life

Each cell of a boolean matrix has 8 neighbors adjacent to it horizontally, vertically, or diagonally. The Game of Life concerns the computation of the next generation boolean matrix.

(0) A 0-cell with 3 neighboring 1-cells becomes a 1.

(1) A 1-cell with 2 or 3 neighboring 1-cells remains at 1.

(2) All other cells remain or become a 0.

There are two main variations on the treatment of cells on the edges of the matrix: (a) the matrix is surrounded by a border of 0s; or (b) cells on an edge are adjacent to cells on the opposite edge, as on a torus.

There is an implementation of life in the dfns workspace and explained in a YouTube video. It assumes a toroidal topology.

``````   life←{↑1 ⍵∨.∧3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}    ⍝ John Scholes

⊢ glider←5 5⍴0 0 1 0 0 1 0 1 0 0 0 1 1,12⍴0
0 0 1 0 0
1 0 1 0 0
0 1 1 0 0
0 0 0 0 0
0 0 0 0 0
life glider
0 1 0 0 0
0 0 1 1 0
0 1 1 0 0
0 0 0 0 0
0 0 0 0 0

{'.⍟'[⍵]}¨ (⍳8) {life⍣⍺⊢⍵}¨ ⊂glider
┌─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐
│..⍟..│.⍟...│..⍟..│.....│.....│.....│.....│.....│
│⍟.⍟..│..⍟⍟.│...⍟.│.⍟.⍟.│...⍟.│..⍟..│...⍟.│.....│
│.⍟⍟..│.⍟⍟..│.⍟⍟⍟.│..⍟⍟.│.⍟.⍟.│...⍟⍟│....⍟│..⍟.⍟│
│.....│.....│.....│..⍟..│..⍟⍟.│..⍟⍟.│..⍟⍟⍟│...⍟⍟│
│.....│.....│.....│.....│.....│.....│.....│...⍟.│
└─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘
``````

Stencil Lives

It has long been known that stencil facilitates Game of Life computations. Eugene McDonnell explored the question in the APL88 paper Life: Nasty, Brutish, and Short [Ho51]. The shortest of the solutions derive as follows.

By hook or by crook, find all the 3-by-3 boolean matrices `U` which lead to a middle 1. A succinct Game of Life then obtains.

``````   B ← {1 1⌷life 3 3⍴(9⍴2)⊤⍵}¨ ⍳2*9
U ← {3 3⍴(9⍴2)⊤⍵}¨ ⍸B  ⍝ ⍸ ←→ {⍵/⍳⍴⍵}

life1 ← {U ∊⍨ {⊂⍵}⌺3 3⊢⍵}    ⍝ Eugene McDonnell
``````

Comparing `life` and `life1`, and also illustrating that the toroidal and 0-border computations can be expressed one with the other.

``````   b←1=?97 103⍴3

x←1 1↓¯1 ¯1↓ life 0,0,⍨0⍪0⍪⍨b
y←life1 b
x≡y
1

g←{(¯1↑⍵)⍪⍵⍪1↑⍵}
p←life b
q←1 1↓¯1 ¯1↓ life1 (g b[;102]),(g b),(g b[;0])
p≡q
1
``````

Adám Brudzewsky points out that life can be terser as a train (fork):

``````   life1a ← U ∊⍨ ⊢∘⊂⌺3 3    ⍝ Adám Brudzewsky
life1b ← U ∊⍨ {⊂⍵}⌺3 3

(life1 ≡ life1a) b
1
(life1 ≡ life1b) b
1
``````

`life1` is an example of implementing a calculation by look-up rather than by a more conventional computation, discussed in a recent blog post. There is a variation which is more efficient because the look-up is effected with integers rather than boolean matrices:

``````   A←3 3⍴2*⌽⍳9
life2 ← {B[{+/,A×⍵}⌺3 3⊢⍵]}

(life1 ≡ life1b) b
1
``````

Jay Foad offers another stencil life, translating an algorithm in k by Arthur Whitney:

``````   life3 ← {3=s-⍵∧4=s←{+/,⍵}⌺3 3⊢⍵}    ⍝ Jay Foad

(life1 ≡ life3) b
1
``````

The algorithm combines the life rules into a single expression, wherein `s←{+/,⍵}⌺3 3 ⊢⍵`

(0) for 0-cells `s` is the number of neighbors; and
(1) for 1-cells ` s` is the number of neighbors plus 1, and the plus 1 only matters if `s` is 4.

The same idea can be retrofit into the toroidal `life`:

``````   lifea←{3=s-⍵∧4=s←⊃+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}

(life ≡ lifea) b
1

``````

Collected Definitions and Timings

``````life   ← {↑1 ⍵∨.∧3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}
lifea  ← {3=s-⍵∧4=s←⊃+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}

B←{1 1⌷life 3 3⍴(9⍴2)⊤⍵}¨ ⍳2*9
U←{3 3⍴(9⍴2)⊤⍵}¨ ⍸ B
A←3 3⍴2*⌽⍳9

life1  ← {U ∊⍨ {⊂⍵}⌺3 3⊢⍵}
life1a ← U ∊⍨ ⊢∘⊂⌺3 3
life1b ← U ∊⍨ {⊂⍵}⌺3 3
life2  ← {B[{+/,A×⍵}⌺3 3⊢⍵]}
life3  ← {3=s-⍵∧4=s←{+/,⍵}⌺3 3⊢⍵}

cmpx (⊂'life') ,¨ '1' '1a' '1b' '2' '3' '' 'a' ,¨ ⊂' b'
life1 b  → 2.98E¯3 |    0% ⎕⎕⎕⎕⎕
life1a b → 1.97E¯2 | +561% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
life1b b → 2.99E¯3 |    0% ⎕⎕⎕⎕⎕
life2 b  → 2.71E¯4 |  -91%
life3 b  → 6.05E¯5 |  -98%
* life b   → 1.50E¯4 |  -95%
* lifea b  → 1.41E¯4 |  -96%
``````

The `*` indicate that `life` and `lifea` give a different result (toroidal v 0-border).

`life1a` is much slower than the others because `⊢∘⊂⌺` is not implemented by special code.

`{+/,⍵}⌺` is the fastest of the special codes because the computation has mathematical properties absent from `{⊂⍵}⌺` and `{+/,A×⍵}⌺`.

The effect of special code v not, can be observed (for example) by use of redundant parentheses:

``````   cmpx '{+/,⍵}⌺3 3⊢b' '{+/,(⍵)}⌺3 3⊢b'
{+/,⍵}⌺3 3⊢b   → 3.13E¯5  |      0%
{+/,(⍵)}⌺3 3⊢b → 2.63E¯2  | +83900% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

cmpx '{+/,A×⍵}⌺3 3⊢b' '{+/,A×(⍵)}⌺3 3⊢b'
{+/,A×⍵}⌺3 3⊢b   → 2.42E¯4 |      0%
{+/,A×(⍵)}⌺3 3⊢b → 2.98E¯2 | +12216% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕

⍝ no special code in either of the following expressions
cmpx '{+/,2×⍵}⌺3 3⊢b' '{+/,2×(⍵)}⌺3 3⊢b'
{+/,2×⍵}⌺3 3⊢b   → 2.92E¯2 |      0% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
{+/,2×(⍵)}⌺3 3⊢b → 3.03E¯2 |     +3% ⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕⎕
``````

That `life` and `lifea` do not use stencil and yet are competitive, illustrates the efficacy of boolean operations and of letting primitives “see” large arguments.

Finally, if you wish to play with stencil a description and a hi-fi model of it can be found here.

Krypto

In the 2016 Year Game, the task was to generate the numbers 0 to 100 using APL primitives and the digits 2 0 1 6 in that order. For example,

``````   20=16
×2016
2⌊016
2+×016
…
``````

This “puzzle of the year” brings to mind Krypto, a game I played many years ago while in grade school.

Krypto

Krypto is a mathematical card game. The Krypto deck has 56 cards: 3 each of numbers 1-6, 4 each of the numbers 7-10, 2 each of 11-17, 1 each of 18-25.

``````   ⎕io←0
DECK ← (3/1+⍳6),(4/7+⍳4),(2/11+⍳7),18+⍳8
``````

Six cards are dealt: an objective card and five other cards. A player must use all five of the latter cards’ numbers exactly once, using any combination of arithmetic operations (`+`, `-`, `×`, and `÷`) to form the objective card’s number. The first player to come up with a correct formula is the winner. The stricter “International Rules” specify the use of non-negative integers only; fractional or negative intermediate results are not permitted.

For example, if the objective card were `17` and the other cards were `2`, `8`, `14`, `19`, and `21`, then a Krypto solution can be as follows. Without loss of generality, we use APL notation and syntax.

``````   8 - 19 + 14 - 2 × 21
``````

In this article we present functions to find all solutions to a Krypto hand.

A Solution

There are a maximum of `!5` permutations of the `5` cards and `4` possibilities in each of the `4` places where an operation can be put, for `(!5)×4*4` or `30720` total possibilities. This number is small enough for an exhaustive approach.

``````deal←{DECK ⌷⍨⊂ 6?≢DECK}

Krypto←{
perm←{0=⍵:1 0⍴0 ⋄ ,[⍳2](⊂0,1+∇ ¯1+⍵)⌷⍤1⍒⍤1∘.=⍨⍳⍵}
a←256⌿5 0⍕⍵[perm 5]
a[;6+5×⍳4]←'+-×÷'[((256×!5),4)⍴⍉(4⍴4)⊤⍳256]
⊣⌸ a ⌿⍨ ⍺ = ⍎⍤1 ⊢a
}

deal ⍬
17 8 19 14 2 21

⊢ t← 17 Krypto 8 19 14 2 21
8 - 19 + 14 -  2 × 21
8 - 19 + 14 - 21 ×  2
8 - 19 - 21 + 14 ÷  2
8 - 14 + 19 -  2 × 21
...
21 +  8 - 19 - 14 ÷  2
21 - 19 -  8 + 14 ÷  2

⍴ t
24 25
``````

Intermediate Steps

The function `perm` is from the Dyalog blog post Permutations, 2015-07-20. `perm n` generates the sorted table of all permutations of `⍳n`.

The local variable `a` in `Krypto` is a `30720`-row character matrix computed from the `5` non-objective numbers. It consists of all `!5` permutations of the `5` numbers interspersed with all `4*4` arrangements of the four operations `+ - × ÷`.

Executing the rows of `a` produces a `30720`-element numeric vector. Comparison of this vector with the objective yields a boolean vector that selects the rows of `a` which are correct formulas.

``````   ⍴ a
30720 25

8↑a
8 + 19 + 14 +  2 + 21
8 + 19 + 14 +  2 - 21
8 + 19 + 14 +  2 × 21
8 + 19 + 14 +  2 ÷ 21
8 + 19 + 14 -  2 + 21
8 + 19 + 14 -  2 - 21
8 + 19 + 14 -  2 × 21
8 + 19 + 14 -  2 ÷ 21
¯5↑a
21 ÷  2 ÷ 14 × 19 ÷  8
21 ÷  2 ÷ 14 ÷ 19 +  8
21 ÷  2 ÷ 14 ÷ 19 -  8
21 ÷  2 ÷ 14 ÷ 19 ×  8
21 ÷  2 ÷ 14 ÷ 19 ÷  8

+/ b ← 17 = ⍎⍤1 ⊢a
24

t ≡ b⌿a
1
``````

We note that use of the `@` operator, new to Dyalog version 16.0, obviates the need to name intermediate results for reasons for syntax. Instead, names are only used to break the code down into more comprehensible components.

``````Krypto1←{
perm←{0=⍵:1 0⍴0 ⋄ ,[⍳2](⊂0,1+∇ ¯1+⍵)⌷⍤1⍒⍤1∘.=⍨⍳⍵}
⊣⌸ ⍺ (⊢(⌿⍨)=∘(⍎⍤1)) '+-×÷'[((256×!5),4)⍴⍉(4⍴4)⊤⍳256]⊣@(6+5×⍳4)⍤1 ⊢256⌿5 0⍕⍵[perm 5]
}

Krypto2←{
perm←{0=⍵:1 0⍴0 ⋄ ,[⍳2](⊂0,1+∇ ¯1+⍵)⌷⍤1⍒⍤1∘.=⍨⍳⍵}
fns  ← '+-×÷'[((256×!5),4)⍴⍉(4⍴4)⊤⍳256]
nums ← 256⌿5 0⍕⍵[perm 5]
⊣⌸ ⍺ (⊢(⌿⍨)=∘(⍎⍤1)) fns ⊣@(6+5×⍳4)⍤1 ⊢nums
}

17 (Krypto ≡ Krypto1) 8 19 14 2 21
1

17 (Krypto ≡ Krypto2) 8 19 14 2 21
1

``````

International Rules

The international rules (intermediate results must be non-negative integers) can be enforced readily:

``````irules←{⍵⌿⍨∧/(i=⌊i)∧0≤i←8 13 18{⍎¨⍺↓¨⊂⍵}⍤1⊢⍵}

irules 17 Krypto 8 19 14 2 21
8 +  2 + 14 ÷ 21 - 19
8 + 21 - 19 - 14 ÷  2
19 -  2 ×  8 - 21 - 14
19 -  2 ÷  8 - 21 - 14
19 -  2 × 14 - 21 -  8
19 -  2 ÷ 14 - 21 -  8
2 +  8 + 14 ÷ 21 - 19
21 - 19 -  8 + 14 ÷  2
``````

It is instructive to look at how the local variable `i` in `irules` is formed:

``````   ⊢ t←17 Krypto 8 19 14 2 21
8 - 19 + 14 -  2 × 21
8 - 19 + 14 - 21 ×  2
8 - 19 - 21 + 14 ÷  2
8 - 14 + 19 -  2 × 21
...
21 +  8 - 19 - 14 ÷  2
21 - 19 -  8 + 14 ÷  2

8 13 18 {⍺↓¨⊂⍵}⍤1 ⊢t
┌─────────────────┬────────────┬───────┐
│19 + 14 -  2 × 21│14 -  2 × 21│ 2 × 21│
├─────────────────┼────────────┼───────┤
│19 + 14 - 21 ×  2│14 - 21 ×  2│21 ×  2│
├─────────────────┼────────────┼───────┤
│19 - 21 + 14 ÷  2│21 + 14 ÷  2│14 ÷  2│
├─────────────────┼────────────┼───────┤
│14 + 19 -  2 × 21│19 -  2 × 21│ 2 × 21│
├─────────────────┼────────────┼───────┤
...
├─────────────────┼────────────┼───────┤
│ 8 - 19 - 14 ÷  2│19 - 14 ÷  2│14 ÷  2│
├─────────────────┼────────────┼───────┤
│19 -  8 + 14 ÷  2│ 8 + 14 ÷  2│14 ÷  2│
└─────────────────┴────────────┴───────┘

⍎¨ 8 13 18 {⍺↓¨⊂⍵}⍤1 ⊢t
¯9 ¯28  42
¯9 ¯28  42
¯9  28   7
¯9 ¯23  42
...
¯4  12   7
4  15   7
``````

(An earlier version of the text appeared as the Jwiki essay Krypto, 2013-07-04.)