teaching by example, data analysis of VIX (volatility option), heat-map, poker simulation, advances in gtk graphics for J7, OpenJ, J and APL meetings, visual interface, general learning, general problem-solving
Meeting Agenda for NYCJUG 20120314 ---------------------------------- 1. Beginner's regatta: we need interesting, practical, domain-oriented J intros: see "Teach J by not Teaching J.pdf" and "Quick and Dirty Data Analysis with J.pdf". 2. Show-and-tell: example of a graphic that should be easier to create: see "Example of type of heat-map.pdf". Some advances in a poker simulation: see "Defining Break Points for Categorizing Hands.pdf" - discuss relations between different facets of agression in poker. 3. News: advances in J7: see "Announcement of gtkwd and grid availability under jconsole.pdf" and "Chess Board in APL and J.pdf". OpenJ: news? See "Google Building Online Chrome Application Shop.pdf", talk about what is a good Android device to target. APL Moot at the YHA Lee Valley Youth Hostel, Cheshunt, on Friday-Sunday, 27-29 April. J Conference Conference - July 23/24 (Monday/Tuesday) 2012 - Toronto: both days start with a continental breakfast at 9am and sessions run to 6pm. The first day ends with an open bar reception and banquet. Register early - before April 1st - and save! See "JSoftware Conference Speakers.pdf The Dyalog 2012 APL programming contest for students is now open! Prizes for participants and referrers! 4. Learning, teaching and promoting J, et al.: see "Visual Interface for J.pdf" and "visualJ4.pdf". General learning and problem-solving: see "Probabilistic Graphic Models - Stanford course by Daphne Koller.pdf", "Viewing the world through mathematical lens.pdf" and "Getting past being stumped by a problem.pdf".
Before the meeting began, our newest member, Scott, had some questions about J. He had looked at some of the introductory materials and was dis-satisfied with the level at which they started - he said they dropped you right into the thick weeds.
He also had a question about whether J was implemented using an LLVM (Low-Level Virtual Machine) framework. The goal of this effort is to provide "a modern, SSA-based compilation strategy capable of supporting both static and dynamic compilation of arbitrary programming languages" as stated on the main website. Scott described it as a layer between language implementations and different hardware and OS platforms. It sounds very relevant to the current dispersion of J onto different platforms, but J continues to suffer the arrows that plague true pioneers. I hastened to point out that it's not a compiled language and it's not parsable within a Backus-Naur scheme of grammar due to its dynamic nature. Perhaps we'll benefit indirectly from LLVM as C compilers are adopted more readily on each new platform.
Once we started the meeting, we looked at Skip Cave's suggestion that J web tutorials focus on teaching math with J as a tool rather than language itself. He pointed to the Khan academy tutorials as an example of compelling instruction that's been proven very popular. He also emphasized the utility of having J tutorials available on cloud servers - this would take advantage of J's recent separation of the front end from the computational engine. Skip also suggested that such a server could be hosted on existing facilities like Amazon at a nominal cost.
We also considered Richard John's suggestion that the followers of reddit would welcome a well-written tutorial on how to solve a problem using J.
These ideas, along with the restriction that a successful tutorial should be organized in small modules, inspired me to take a stab at this genre with a one-page example of using J code to download financial data from the web and analyze it. This is presented as the "Quick and Dirty Data Analysis with J.pdf" paper - also here.
We start with motivation from a Wall Street Journal article about the VIX - the volatility index. The article makes some verifiable claims about the level of this index, so one of the challenges is to check these claims.
We start by showing where we find the data - on Yahoo Finance (finance.yahoo.com) - and assume we downloaded the entire daily price history of the VIX into a .csv file. The J code, commented fairly extensively, shows how we might read this file into some variables and select a portion of it (dates and adjusted closing prices) to analyze. We show different things we can figure out from the data, including verifying the claims in the article. Along the way, we also illustrate alternate J formulations to accomplish the same task, as well as a brief example of deriving a tacit expression from explicit code.
In looking at the graphic in "Example of type of heat-map.pdf" (below), we see how a large amount of data can be shown in a manner that lets us draw some useful inferences from it quite easily. Even looking at the black-and-white version in our printed materials, it's easy to see that three of the items stand out from the others as dark vertical lines across the entire time period. These three items represent the industries "multi-utilities", "gas utilities", and "electric utilities", which are obviously closely related to each other. We also see a cross-industry effect of some kind starting at the beginning of 2011 and persisting until near the end of that year.
This kind of graphic, which was generated by viewmat, is obviously very helpful. However, the difficulty with the example shown is that all the labels and the key had to be inserted into the image manually. It's not too hard to see that this could be handled more programmatically but we don't presently have an easy and flexible way to insert arbitrary text into an image. Ric, joining us online, mentioned that R is able to generate graphics like this fairly easily. It remains a future project for when we get more comfortable with the capabilities of the GTK-based graphics.
Our next section, "Defining Break Points for Categorizing Hands.pdf", focused on a discussion of some preliminary work on a poker simulation. Like the earlier section on "quick and dirty data analysis", this material could be suitable for a tutorial on J. The paper explains work done to build a foundation for parameterizing qualitative features like "conservative" or "aggressive" play. We show how we develop some simple tools to analyze the results of many simulated games and discover some features of the data extracted from these simulations.
Our regular "Advanced Topics" section was replaced by "News" this month as there was a lot to talk about. There has been a lot of work done on various facets of "OpenJ" and there are a number of upcoming J-related events for which we should be planning.
Perhaps the most exciting recent work in OpenJ was Michael Dykman's completion of his initial Android port of jconsole. As we were discussing issues around this and the desirability of porting J to other platforms like Chrome, Zach downloaded the new Android port onto his phone and showed us all the working version of J; the JHS version also runs on his phone. There was talk of a version of Ubuntu Linux that runs on Android as well as the availability of Android on Intel platforms. Intel appears to be targeting their Atom chip for the Android implementation as this is a low-power chip more suitable for mobile devices than are their main products.
One issue that came up was, with the multitude of platforms and directions for J to evolve toward, the efforts on the language are getting somewhat diffuse: we're going in many directions at once. This is probably a good thing in general but, given the smallness of our community, it slows the gains in any particular direction. In spite of this, we are seeing progress on a number of areas, not the least of which is getting a full set of GTK features accessible from J - both in the console and JHS versions. It looks like there's growing grid functionality which leads to a number of interesting possibilities.
We talked briefly about the upcoming JSoftware conference in Toronto (July 23-24). A few of us in the room are either attending or speaking at this much-anticipated event, though some of us have only a vague notion about what we might be speaking.
There's also an APL moot next month - April 27-29 - just outside of London. I may be the only J person going to this but it promises to be very enjoyable and intellectually stimulating.
There are also conferences by STSC (April 22 -24 in Jersey City NJ) and Dyalog (Elsinore, Denmark, from Sunday, October 14) later this year, as well as the Dyalog APL programming contest, but we didn't really talk much about these other than to note the Dyalog contest has prizes for non-students.
Learning, teaching, and problem-solving
Moving on, we discussed the idea of a visual interface for J - what this might mean and how it might look - by considering a few existing visual interfaces geared toward programming largely by assembling visual icons. We looked at the visual formula builder for Clarifi - a CapitalIQ product on which I work - as well as a web-based tool called Waterbear and an example (sample below) of a visual representation of J sent to us by Ronan Reilly.
The visual metaphor has some attraction but we all seemed to be in agreement that the compelling case for it has yet to be made. It's hard to judge fairly these novel visual representations against a notation with which many of us are quite familiar. By comparison to existing J, the visual systems seemed somewhat bulky and not sufficiently well-ordered to be unambiguous aids to understanding: it appears that any of pictures could be drawn in quite a few different ways. A typical visual representation seems to have a tendency to grow across the screen rather quickly compared to the more compact notation it purports to represent.
We also discussed an online Stanford course on Probabilistic Graphical Models and the difficulty of understanding much of what has been written about Bayesian networks.
The meeting ran later than usual but we didn't have time to get to the more general material on teaching mathematics to fairly young children (and their ability to understand these concepts) and research on techniques for getting around mental blocks to problem-solving.
-- DevonMcCormick 2013-12-13 16:30:44