** Who am I? Hello, my name is Joakim Hårsman and I work at Profdoc Care. We make Take Care, a medical record system. Take Care keeps track of all data related to the patient's care: lab results, case notes, referrals, dictated sound files, images, scanned paper documents, the bed and room the patient's in, admission and discharge, payment at the cash registry and so on. [screen shots of various modules or maybe one shot with a potpourri] ** TakeCare ...is a fairly large system: [picture] - 1.8 million medical records - 30 000+ (mostly) happy users - 300 000 lines of APL (roughly equivalent to around a gazillion lines of Java) - 200 transactions per second during office hours - 1 TB+ of data (not really big by today's standard) - Most data in component files (some stuff, e.g images and sound are in plain binary files) - All in Dyalog APL I'm not going to talk much about TakeCare, I just want to give an idea of what we do. If you want to know more, grab a hold of me at lunch. ** Profdoc Care - small (but growing) Founded by Peter Pichler and Martin Williamson in 1995 or so (then under as different name, bought by Profdoc in 2002), the core of TakeCare was developed by them with add ons by two external consultants. I joined in 2005 and was the first one that didn't know APL to start hacking on TakeCare. Since then we've hired eight more developers and none had any experience with APL when they started. We've grown a lot the last few years, both in number of people and the size of our code base. [graph of LOC] Of course, more lines of code really isn't a good thing, but we're working on making that graph go down instead of up. We do have more features as well, so at least we got something out of writing all those lines of code. By volume, the code mostly deals with GUI and robustness issues. Not very much in the way of advanced financial linear algebra or anything like that. ** Common misconceptions about APL Obstacles to overcome to grow your APL business. People that don't know APL think that: - It's a "write only language" and is impossible to understand - APL systems are obsolete "legacy" systems ("legacy systems" roughly means "systems that actually work" as far as I can tell) that need to be replaced. - It's impossible to hire developers for APL work - It cannot interface with other systems People that do know APL think that: - "Normal" programmers have been damaged by learning inferior languages, it will take them years to learn APL. - Developers using other tools and programming languages use them because they don't know anything better (i.e. APL) - The practice of storing source code in text files, using version control systems, having a bug tracker, or other mainstream development practices is not needed for APL, because it has a better way. Some of these are plain uninformed (e.g that APL is "write only" and cannot interface with anything else), but I'm going to talk about the ones I feel are important to debunk: - It's impossible to hire APLers, you cannot grow an APL business. - If you hire ordinary programmers, not APLers, they have been damaged by learning inferior languages, and it will take them years to learn APL. - Developers using other tools and programming languages do not understand the awesome power that is APL, that's why they use inferior tools. - The practice of storing source code in text files, using version control systems, having a bug tracker, or other mainstream development practices is not needed for APL, because it has a better way. ** Recruiting We *have* grown successfully, none of the people we've hired knew APL when they came to us. So how did we do it? *** Don't hire APL programmers The first thing to realize is that you probably shouldn't be looking to hire people experienced in APL. Not that there's anything wrong with them, they are just very hard to find, because there aren't very many around. Instead, focus on hiring good programmers, and they will have no problem picking up APL. It turns out that learning a new programming language takes very little time compared to getting to grips with a large complex software system and the problem domain. If your system is small and simple and solves easy problems, then there's no need to whine about it being in APL, just rewrite it quickly in whatever language your current developers prefer, and then your problem with APL goes away. The question then becomes how to hire good programmers (of course, there are other qualities beside programming skill that matter, but those are independent of the programming language used, so I won't talk about them here). *** Filter out bad programmers early To make it easier to find good programmers, it helps to weed out the bad ones as early as possible. Candidates fill in a small form on our website to send in their application, this form contains questions that we've found useful in weeding out bad developers. There is a very simple programming question that requires candidates to add a couple of numbers in a loop and supply the answer. A surprising amount of people, around 60%, get it wrong. The language used is Python, mainly because it's easy to read even if you don't know it, and the interpreter is a free download. The problem is very easy, because it's purpose is not to find good programmers, but to eliminate bad ones. We ask if they've done any programming in their free time, as a hobby. In our experience, virtually all good programmers say yes. We also ask a question about usability, because that is very important to us. Finally, there are some traditional questions: why we should hire them, why they're applying and a place to attach their resume. This weeds out about 70% of applicants. *** Have candidates write code during the interview The interview consists of three parts: - A programming test where candidates solve three problems in a subset of APL - A design and usability part where they critique a user interface - A more general traditional interview with Martin (our CEO) *** Results It works great! New hires certainly present pretty verbose solutions at first, but that can be fixed during code review. If you hire great developers they will dazzle you with great solutions and features, no matter what language you give them (as long as it's somewhat appropriate for the domain, e.g you don't use Fortran to code a Windows app). The idea that hiring APLers is impossible is nonsense, because it's irrelevant. Don't try to find APLers (they are very hard to find), hire good programmers and they will do just fine with APL. You probably need at least one APL guru to answer questions and give advice, but that role can probably be filled by a consultant if you don't have any APL gurus handy. In conclusion, hiring is not a problem as long as you hire good programmers (of course, other skills beside programming are important as well, but they are independent of the programming language). If you cannot hire good programmers you have a bigger problem, unrelated to APL. ** What we changed [keep this very short] Of course, growing isn't all rainbows and lollipops. Managing a big, organically grown code base, parts of which is ten years old, and remaining agile and adaptable, requires effort. We've had to change some things. *** Better documentation Documentation was spotty, especially on bigger picture stuff like the overall architecture, and why it was designed that way. We installed a wiki and started documenting there. This is great because it's really easy to just jot down stuff without having to prepare some official looking documentation. It's also easy to see who has written what in case you have questions. No one ever locks some document that you want to update on a wiki, so you never have to roam the office to find them so they can close the document you want to edit. This doesn't mean you should become all enterprisey and spend all your time documenting silly stuff that is apparent from the source code anyway. Document why, not how, and focus on big picture stuff that is hard to understand from source code. [show graph of comments here?] Worked pretty well, still lots of work to do. *** Fewer releases Because there are more of us, the pace of change is higher. We used to do one or two releases a week, but as we grew, we spent an inordinate amount of time managing and testing the releases. Now we release every 3-4 weeks and that works much better. *** More code review We spend much more time on testing and code review than we used to. Everyone used to be an expert, that's no longer the case. More review and testing is also needed because the system is much bigger and therefore the cost of bugs is much higher. ** Better tools I'm not sure what you do, but we've never saved workspaces to disk and then used those in production. We write code interactively of course, but it's saved in component files and those are then used to build workspace files that are used to load code dynamically in production. In what I gather is typical APL fashion, we used our own source control system based on component files, our own bugtracker, and a diff tool written in APL. Although I haven't known APL for very long, it seems to me that APL code tends to live in it's own insular little universe, cut off from the outside world. In my not so humble opinion, as a "normal" programmer that have learned APL after knowing other languages, all the "normal" programmers using common source control tools, editing their code in Emacs and vi, and using all the other common development tools, aren't misguided and missing the point. Some of those tools are actually pretty good. *** Free your code The fact that APL code is stored in binary files makes it hard to integrate with common programming tools, since they generally assume that your code is stored in text files. If you have the ability to store code in text files you open up to using these tools. The APL character set used to complicate things, but with Unicode and UTF-8, everything mostly just works. Some examples: **** Diff and merge We used to use a custom tool built in APL for diffing source code. It was ok, but would get easily confused and didn't support merging. I wrote a simple utility function that would write out the functions being compared to text files, so you could use common diff tools. We ended up using WinMerge, which is an open source diff tool for windows. [demo or screen shot of WinMerge] This was much better, but there are still some nagging issues: - Fine differences aren't computed correctly because WinMerge doesn't know how to tokenize APL. - No syntax highlighting: harder to read when reviewing changes and easy to miss syntax errors. - No three way merge I figured there must be some diff tool that allows you to enter some regexps for syntax highlighting. I looked but didn't find anything, even expensive commercial diff tools couldn't do it. Then i remembered Emacs. Emacs is the worlds premier text editor building platform and extremely customizable. And I remembered it supported diffing and merging. So I wrote a couple of hundred lines of Emacs lisp and ended up with a pretty good tool for diff, merge and review: [demo or screen shot of ediff] - Fine differences are correct because I wrote nine lines of Emacs lisp that tokenizes APL in a passable manner for diffing - There's some basic syntax highlighting based on regular expressions. Not completely correct but enough to make it easier to review changes. **** Source control When you have more developers, you might be looking at some code and wondering who made it look that way, and when. The component file storage system for code didn't really support this. But if you store your code in text files, you can use any off the shelf tool for source control. We use Subversion, which is probably the most widely used one, but you can use anything really. We tried a couple of different ones before settling on Subversion, and might switch to Mercurial in the future. [demo of TortoiseSVN] This makes debugging much easier (why is this line of code like this? when did it change?) and made having a separate branch of the code for our Malaysia office practical. Because we're using a standard source control system, plenty of things are already integrated with it. Emacs for example. [demo of annotate in emacs] **** Automation Releasing a new version or creating a build used to require a bunch of manual steps. This became way to error prone as the system grew and most developers didn't know every intimate detail. Now building and releasing are done by tools, it's harder to screw up and you can easily add tests and sanity checks to the tools. The tools are written in APL, which is nice because it makes introspection of the code easier. You could probably use standard build tools if you wanted to. The build tool is written in APL, so that doesn't use off the shelf build tools. For us, that was the only reasonable way to do it since our builds are based on a bunch of meta data stored in APL functions. It's also nice to have the same build tool be able to use both kinds of storage, component and text files. I'd very much like to hear about your build setup, if you do use one. ** Conclusion 1. Hire great programmers 2. Get your source code into Emacs 3. ??? 4. Profit! I might need to be more explicit in step three, but it certainly seems like a decent plan. Seriously, there really is nothing stopping you from growing your APL business. Recruiting really isn't a problem, and if you just open up your APL code a bit to the outside world, you can benefit from thousands of great tools used by other developers. As a bonus, it makes it easier for new hires to dive into the code base since they can use familiar tools.