NYCJUG/2008-10-14

JDB, interactive graphical interface

Location

Heartland Brewery, 34th and 5th, NYC

Meeting Summary

We started off by talking about some figures on the biggest daily changes in the Dow Jones Industrial average and how some graphs of this might be improved and how this might be incorporated into the interactive graphics tool we're planning. Also, we talked about the new J database JDB and how we might influence and aid this effort. Finally, we talked about how these two efforts might relate to each other.

Agenda for NYCJUG of 20081014

1. Introducing J: publicizing the many packages available.

2. Show-and-tell: JDB introduction - what would be useful to have?
What would help support time-series data? 

3. Advanced topics: Interactive Graphics: what features should this have?
See "Samples of Some Existing Plotting Packages.doc" for examples of
what is already out there.

4. Learning and teaching J: frustrations in finding things that you
know are there.

            +.--------------------+.

To sum up: it is wrong always, everywhere, and for anyone, 
to believe anything upon insufficient evidence.
  - William Kingdon Clifford, "The Ethics of Belief"

Proceedings

To start the meeting off, we considered a timely topic much on everyone's mind these days by looking at a list I'd prepared of the biggest daily moves in the Dow Jones Industrial average.

The DJI is not the most widely used index these days but people are familiar with it because it's been around a long time. In fact, the data series I downloaded from Yahoo! Finance begins in October of 1928. This makes it almost exactly 80 years old, which is a nice number of years to consider for a number of reasons. One thing that I was looking at was how the number of big "up" days compares to the number of big "down" days and if this relation has changed over time in any easily-characterizable way.

One of the convenient things about this 80-year period is that it divides evenly into four 20-year periods which more-or-less coincide with important eras in the investing world. The first 20 years, from 1928 through late 1948, covers the Great Crash, the Great Depression, and World War II. The second period covers the post-war era through the culturally seminal year of 1968. The third period covers the great bear market of the early 1970s and the great crash of 1987 (which is at the very top of the list for a single day's move.) The most recent period covers the post-'87-crash, the dot-com boom and bust, to the recent turbulence.

One interesting thing to note is that the years 1929-1933 still dominate the top twenty. Another thing to notice is that there are no days in the top twenty for the years between 1939 and 1987.

Here we show the distribution of the largest daily changes upward (darker bar) versus those downward (light bar).

They all look somewhat similar until you pay close attention to the scales on the bottom of the graphs which are quite different. However, since each individual histogram is scaled according to its own data, the graphs of these four periods are not to be simply compared to each other - they differ more than first appears.

Here we see a crude attempt to use a common scale across all four by forcing the same minimum and maximum X-value onto all the charts.

This highlights the difficulty of doing this well for a few reasons. For instance, though the X-scale is the same across all four periods, the Y-scale is not. Even more importantly, the way I achieved even the minimal commonality of the maximum X-value was by cheating: I added spurious minimum and maximum values to the three series lacking the true minimums and maximums (from the 1948-1968 and 1968-1988 graphs, respectively), then manually erased the very small spurious bars from each graph after it had been rendered as a picture.

All of this points to some fairly obvious ways of better graphing that are very hard to accomplish with existing packages. In fact, I had first noticed this problem when generating graphs of multiple, related series with S-Plus, a language with highly-regarded graphing capabilities. This language is virtually the same as the freely available "R" language to which J has an interface. In fact, Thomas, who started this interactive graphics initiative, had mentioned that they use this interface specifically to generate graphs from J which it cannot do well on its own.

In fact, you may notice that these charts are slightly out-of-synch with the table of numbers because I re-ran the numbers subsequent to the big market moves in October, but have not re-done the charts. That's because it's easy to re-run the numbers but time-consuming to re-do the charts. This difficulty of updating charts was another motivation for Thomas to start work on the interactive graphics project and it's a common problem if you work with a lot of charts.

Beginner's regatta

There are getting to be quite a few packages available in J. Here is a list of them currently:

We need to work on using some of these, giving feedback to their creators to improve them, and publicize them. This brings us to our next topic which is one of these packages: JDB.

Show-and-tell

I've done some preliminary work with JDB. The first thing to do is to load up some different kinds of datasets to see how well it handles them. I have three datasets in mind for this preliminary investigation: the Netflix Challenge data, some options data, and some data on commodities.

These reflect my own interests and data I have readily available. Each of the three should test a different facet of JDB. The Netflix dataset is fairly large and should test JDB's capacity. The options data is a fairly complicated example of time-series data. Even today, the major databases do not have time-series handling built in - each user has to cobble it together ad-hoc.

A summary of my experiences to-date with JDB can be found here: DevonMcCormick/JDBWithNetflixChallengeData.

Advanced topics

Learning and teaching J

Scan of Meeting Notes

CategoryWorkInProgress