{"id":364,"date":"2013-09-02T22:23:08","date_gmt":"2013-09-02T22:23:08","guid":{"rendered":"http:\/\/cto.dyalog.com\/?p=364"},"modified":"2014-07-21T15:48:50","modified_gmt":"2014-07-21T15:48:50","slug":"apl-controlled-robot-performs-death-defying-stunts-using-picam","status":"publish","type":"post","link":"https:\/\/www.dyalog.com\/blog\/2013\/09\/apl-controlled-robot-performs-death-defying-stunts-using-picam\/","title":{"rendered":"APL-Controlled Robot Performs Death-Defying Stunts Using PiCam"},"content":{"rendered":"<p>Regular readers will remember my whining about the poor precision of both infra-red and ultrasonic sensors. But today,\u00a0the Raspberry Pi \/ Dyalog APL &#8211; controlled &#8220;DyaBot&#8221; was observed driving on a dinner table &#8211; where the slightest navigational error could mean a 3-foot plunge and certain death! How can this be?<\/p>\n<iframe loading=\"lazy\"  id=\"_ytid_75913\"  width=\"584\" height=\"329\"  data-origwidth=\"584\" data-origheight=\"329\"  src=\"https:\/\/www.youtube.com\/embed\/yZd3UOdtnNw?enablejsapi=1&autoplay=0&cc_load_policy=0&iv_load_policy=1&loop=0&modestbranding=0&rel=1&fs=1&playsinline=0&autohide=2&theme=dark&color=red&controls=1&\" class=\"__youtube_prefs__  no-lazyload\" title=\"YouTube player\"  allow=\"autoplay; encrypted-media\" allowfullscreen data-no-lazy=\"1\" data-skipgform_ajax_framebjll=\"\"><\/iframe>\n<p>The answer is: the\u00a0&#8220;<a title=\"PiCam at raspberrypi.org\" href=\"http:\/\/www.raspberrypi.org\/camera\" target=\"_blank\">PiCam<\/a>&#8221; finally arrived last week!!!<\/p>\n<p><span style=\"color: #000000; font-weight: bold;\">Capturing Images with The PiCam<\/span><\/p>\n<p>I no longer need to complain about sonar beams being up to 30 degrees wide: the PiCam has a resolution of up to 1080&#215;1920 pixels! So all we need is some software to interpret the bits&#8230;First, we\u00a0capture images using the &#8220;raspistill&#8221; command:<\/p>\n<pre>raspistill -rot 90 -h 60 -w 80 -t 0 -e bmp -o\u00a0ahead.bmp<\/pre>\n<p>The parameters rotate the image 90 degrees (the camera is mounted &#8220;sideways&#8221;),\u00a0set the size to 60 x 80 pixels (don&#8217;t need more for navigation), take the picture immediately, and store the output in a file called &#8220;ahead.bmp&#8221;. (Documentation for the camera and related commands can be found\u00a0<a title=\"RaspiCam Documentation at raspberrypi.org (PDF)\" href=\"http:\/\/www.raspberrypi.org\/wp-content\/uploads\/2013\/07\/RaspiCam-Documentation.pdf\" target=\"_blank\">on the Raspberry Pi web page<\/a>.)\u00a0Despite the small number of pixels, the command takes a full second to execute &#8211; anyone who knows a way to speed up the process of taking a picture, please let me know!<\/p>\n<p>In the video, each move takes about 3 seconds, this is simply because each cycle is triggered by a browser refresh of the page after 3 seconds. Capturing the image takes about 1 second, the &#8220;image analysis&#8221; about 40 milliseconds, so we could be driving a lot faster with a bit of Javascript on the client side (watch this space).<\/p>\n<h2>Extracting BitMap Data<\/h2>\n<p>Under Windows, Dyalog APL has a built-in object for reading BitMap files, but at the moment, the Linux version does not have an equivalent. Fortunately, extracting the data using APL is not very hard (after you finish <a title=\"BMP File Format - Wikipedia\" href=\"http:\/\/en.wikipedia.org\/wiki\/BMP_file_format\" target=\"_blank\">reading about BMP files on WikiPedia<\/a>):<\/p>\n<pre><code class=\"language-apl\">tn\u2190'ahead.bmp' \u2395NTIE 0     \u235d Open \"native\" file\r\n(offset hdrsize width height)\u2190\u2395NREAD tn 323 4 10 \r\n                           \u235d \u2191\u2191\u2191 Read 4 Int32s from offset 10\r\ndata\u2190\u2395NREAD tn 80 (width\u00d7height\u00d73) offset \u235d Read chars (type 80)\r\ndata\u2190\u2296\u2395UCS(height width 3)\u2374data \u235d Numeric matrix, reverse rows<\/code><\/pre>\n<p>The above gives us a 60 by 80 by 3 matrix containing (R,G,B) triplets. This code assumes that BMP file is in the 24-bit format created by raspistill; I will extend the code to handle all BMP formats (2, 16 and 256 colours) and post it when complete &#8211; but the above will do for now.<\/p>\n<h2>Where&#8217;s the Edge?<\/h2>\n<p>At Iverson College last week, I demonstrated the DyaBot driving on a table with a green tablecloth, using code which compared the ratio between R,G,B values to an average of a sample of green pixels from one shot. Alas, I prepared the demo in the morning and, when the audience arrived, there was much more (yellowish) light coming in through the window. This changed the apparent colour so much that the bot decided that the side of the table facing the window was now unsafe, and it cowered in the darkness.<\/p>\n<p>Fortunately, I was talking to a room full of very serious hackers, who <a title=\"Kernel Image Processing - Wikipedia\" href=\"http:\/\/en.wikipedia.org\/wiki\/Kernel_%28image_processing%29\" target=\"_blank\">sent me off to Wikipedia to learn about &#8220;kernels&#8221; as a tool for image processing<\/a>. Armed with a suitable edge detection kernel, I was able to test this new algorithm on a dozen shots taken at different angles with the PiCam. Each pair of images below has the original on the left, and edges coloured red on the right. Notice that, although the colour of the table varies a lot when viewed from different directions &#8211; especially when there is background glare &#8211; the edges are always correctly identified:<\/p>\n<p><a href=\"http:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Edge-Detection-Examples.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-379\" src=\"http:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Edge-Detection-Examples.png\" alt=\"Edge Detection Examples\" width=\"793\" height=\"423\" srcset=\"https:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Edge-Detection-Examples.png 793w, https:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Edge-Detection-Examples-300x160.png 300w, https:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Edge-Detection-Examples-500x266.png 500w\" sizes=\"auto, (max-width: 793px) 100vw, 793px\" \/><\/a><\/p>\n<p>Except for the image at the bottom left, where the bot is so close to the edge that the table is not visible at all (and the edge is the opposite wall of the room, ten feet away), we seem to have a reliable tool.<\/p>\n<h2>The PiServer Page<\/h2>\n<p>The code to drive the bot is embedded in a &#8220;PiServer&#8221; (MiServer running on a Rasperry Pi) web page. Each refresh of the page takes a new picture, extracts the bits, and calls the\u00a0main decision-making function.\u00a0The suggested action (turn or drive straight ahead) is displayed in the form, and the user has the choice of pressing OK to execute the command, pressing the &#8220;Nah&#8221; button to take a new photo and try again (after moving the bot, changing the lighting in the room, or editing the code). There are also four &#8220;manual&#8221; buttons for moving the Bot. After testing the decision-making abilities of the code, brave programmers press the &#8220;Auto&#8221; button, allowing the robot to drive itself without waiting for confirmation before each command (see the video at the start of this post)!<\/p>\n<p><a href=\"http:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Robot-Views-Table-at-Dyalog-House.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-381\" src=\"http:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Robot-Views-Table-at-Dyalog-House.png\" alt=\"Robot Views Table at Dyalog House\" width=\"844\" height=\"765\" srcset=\"https:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Robot-Views-Table-at-Dyalog-House.png 844w, https:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Robot-Views-Table-at-Dyalog-House-300x271.png 300w, https:\/\/www.dyalog.com\/blog\/wp-content\/uploads\/2013\/09\/Robot-Views-Table-at-Dyalog-House-330x300.png 330w\" sizes=\"auto, (max-width: 844px) 100vw, 844px\" \/><\/a><\/p>\n<h2><span style=\"color: #000000; font-weight: bold;\">The Code<\/span><\/h2>\n<p>The central decision-making function and the kernel computation function are listed below. The full code will be uploaded to the MiServer repository on GitHub, once it is finally adjusted after I find a way to attach the PiCam properly, rather than sticking it to the front of the Bot with a band-aid!<\/p>\n<pre><code class=\"language-apl\">\u2207 r\u2190StayOnTable rgb;rows;cols;table;sectors;good;size;edges\r\n \u235d Based on input from PiCam, drive at random, staying on table\r\n \u235d Return vector containing degrees to turn and \r\n \u235d               #seconds to drive before next analysis\r\n\r\n (rows cols)\u2190size\u21902\u2191\u2374rgb\r\n \u235d First detect edges in each colour separately\r\n edges\u2190EdgeDetectAll\u2218AK\u00a8\u2282[1 2]rgb\r\n \u235d Call it an edge if any r, g or b result is &gt;75% of original\r\n edges\u2190\u2228\/(\u2191[0.5]edges)&gt;0.75\u00d71 1\u2193\u00af1 \u00af1\u2193rgb\r\n \u235d Look for lowest edge in each column \r\n table\u2190+\u233f\u2227\u2340~\u2296edges\r\n \u235d Divide into 3 equally sized sectors (left, centre, right) \r\n sectors\u2190((\u2374table)\u2374(\u2308(\u2374table)\u00f73)\u21911)\u2282table\r\n \u235d Find the LOWEST edge in each sector\r\n good\u2190\u230a\/\u00a8sectors\r\n\r\n :If good\u2227.&gt;15 \u235d More than 15 pixel rows of table in all sectors\r\n    r\u21900,0.1\u2308(good[2]-15)\u00f725 \u235d Carry straight on for a while\r\n :Else \u235d Some sectors have less than 15 pixel rows\r\n    :If 0\u22601\u2191previous \u22c4 r\u2190previous \r\n        \u235d Once started turning, keep turning the same way\r\n    :Else \r\n        r\u2190((1+&gt;\/good[1 3])\u228345 \u00af45),0 \u235d Turn in \"best\" direction\r\n    :EndIf\r\n :EndIf\r\n previous\u2190r \u235d Remember last turn for next decision\r\n\u2207<\/code><\/pre>\n<p>The function AK (Apply Kernel), and the kernel are defined as follows:<\/p>\n<pre><code class=\"language-apl\">\u2207 r\u2190kernel AK data;shape\r\n\u00a0 \u00a0shape\u2190\u2374kernel\r\n\u00a0 \u00a0r\u2190(1-shape)\u2193\u2283+\/,kernel\u00d7(\u00af1+\u23731\u2191shape)\u2218.\u2296(\u00af1+\u23731\u2193shape)\u2218.\u233d\u2282data\r\n\u2207\r\n\r\n EdgeDetectAll \u235d Our 3x3 kernel\r\n\u00af1 \u00af1 \u00af1\r\n\u00af1\u00a0 8 \u00af1\r\n\u00af1 \u00af1 \u00af1<\/code><\/pre>\n<p>Note the similarity of AK to the <a title=\"Game of Life in APL - YouTube\" href=\"http:\/\/www.youtube.com\/watch?v=a9xAKttWgP4\" target=\"_blank\">APL Code for Conway&#8217;s Game of Life<\/a>!<\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Regular readers will remember my whining about the poor precision of both infra-red and ultrasonic sensors. But today,\u00a0the Raspberry Pi \/ Dyalog APL &#8211; controlled &#8220;DyaBot&#8221; was observed driving on a dinner table &#8211; where the slightest navigational error could &hellip; <a href=\"https:\/\/www.dyalog.com\/blog\/2013\/09\/apl-controlled-robot-performs-death-defying-stunts-using-picam\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,12,11,9,5],"tags":[],"class_list":["post-364","post","type-post","status-publish","format-standard","hentry","category-apl-language","category-cto","category-miserver","category-raspberry-pi","category-robots"],"_links":{"self":[{"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/posts\/364","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/comments?post=364"}],"version-history":[{"count":38,"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/posts\/364\/revisions"}],"predecessor-version":[{"id":510,"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/posts\/364\/revisions\/510"}],"wp:attachment":[{"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/media?parent=364"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/categories?post=364"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dyalog.com\/blog\/wp-json\/wp\/v2\/tags?post=364"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}