Rimfire Project

The automation of agency cash betting & payout for the
Victorian Totalizator Agency Board (TAB)

By Marcel Dayan - 20 February 2008

Acknowledgement

As these events occurred almost 40 years ago, I am indebted to a number of colleagues who have helped me on details and to compile this narrative. In particular, I would like to thank Stuart Broad and Phil Stokes from CDA and Tom Daniel and Olaf Sid from TAB who provided valuable inputs and Geoff Hipwell, Ron Bird and Alison & Adrian Bryant who sent photos and scanned articles from "Between Ourselves" newsletters.
Roger & Andrea Taylor also kindly provided photos and articles from 'Tabloid' newsletters (Andrea was the TAB editor of the 'Tabloid' up to 1975).

Discussions at the Friday lunch with John O'Neil and Tony Bell were also very helpful in fine tuning some of the details

The Project

I joined Control Data Australia's Sydney office in late February 1968 working as a pre-sales analyst in process control applications and was just settling into this environment when I was asked two questions by my Melbourne manager, George Karoly:

1. Did I have any moral objections to working on a horse racing/gambling project ?
(CDC was a very conservative mid-west company)
2. Did I want to go to Minneapolis for about 3 months?

Being single and foot loose, I asked what the catch was and was told that it would mean transferring to Melbourne office on my return. On accepting the assignment, I moved to Melbourne in early May 1968 and found myself with 3 CDC engineers, Bill Criego (leader), Milt Spieler and Orrin Butterfield undergoing a 4 week intensive course to find out how the Victorian TAB operated their retail network. This included a full week's "agent course" as well as on-course visits, operating the telephone betting terminals and lectures by TAB management.

Victorian TAB had already installed two CDC 3100 computers that automated the telephone betting function and the amalgamation of agent's collations via the "Plessey" keysets - called the CARBINE System (Computer Automated Real-time Betting Information NEtwork). The RIMFIRE System (Remote Input Machines For Investments on Racing Events) was to automate the retail cash betting (and paying) process.

The 10 person team that was assembled to design this system consisted of the 3 CDC engineers, myself from CDA and 6 representatives from the Victorian TAB. The TAB team consisted of Charles Scorgie (Data Processing Manager), Bart Godwin (his deputy), Liz Allison (his secretary), Barry Blair (software), Terry McAuley (hardware) Bill Criego (CDC) and Lyn Newman (operations). Subsequently Lyn married Mike Howard from CDA.

The team was scheduled to assemble in Minneapolis mid-June 1968 and I particularly remember my trip over to the US with Milt Spieler (which was via HK, Israel and UK) for the announcement of Bobby Kennedy's assassination on June 6th just as we were landing in HK.

The computer system consisted of 5 x CDC 1700 modules (four modules were on-line with the fifth module used as back-up) each with 32 K words of 16 bit memory , a 256K word drum (no disk) and a dual multiplexer front-end that interfaced 128 lines communicating at 150 Bits per second. Of these, 115 were for communicating to "live" branches and agencies, the rest to connect to the Carbine system, other modules and spares. There were 2 RIOT's on each line, with each polled in turn (so as to have one RIOT receiving while the other one was transmitting) and so each RIMFIRE module was capable of handling 230 RIOTS.

1700 Computer Reference Manual

Because of the high cost of computer storage, it was not viable to store the individual bet details and so payout was a two pass operation accomplished by re-keying the original bet details (together with a hashed number incorporating these details plus the date) and comparing this to the winning result. (This lack of security eventually required a sequence number to be incorporated in the hashed number, which was then matched in a non real time mode to stop duplicate payouts - see Subsequent Happenings.)

The TAB team returned home mid September, but I stayed on in Minneapolis with the rest of the CDC team, refining the design until mid-December 1968 when Jim McGeorge insisted I return to help him answer the RFP from NSW TAB for an automated system. Following the TAB team's return, Ken Davis - TAB General Manager appointed Bart Godwin as Data Processing Manager in place of Charles Scorgie who resigned.

For most of 1969 I was then seconded to the CARBINE II project (upgrading the CDC 3100s to CDC 3300s and expanding the number of accounts that could be handled with the new CRT Information Electronics Telephone Betting Terminals). This project also included an interface to the RIMFIRE modules for results transmission and to collect the collations.

In May 1969, account rep Tony Blackmore (together with Trevor Robinson and John O'Neil in the background) negotiated the RIMFFIRE contract between TAB and CDA for the 5 x CDC 1700 computers and 1000 RIOT betting terminals at a total value of around $6 million. This was priced on the hardware only; the software development was thrown in for free. The software would initially be developed by CDC in Minneapolis (with the help of 2 CDA programmers) and they would then complete the work in Australia commencing August 1970 and go on-line May 1st 1971. The CDC programming leader was Arnie Hochhalter and the two CDA programmers who joined the Minneapolis team were Stuart Broad and David Gross.

With the setting up of the Australian Systems Division in January 1970, I was appointed Project Manager for the RIMFIRE Project (initially reporting to Ben Louw and subsequently to Ron Bird) and visited Minneapolis to monitor progress in late January. This was then followed by a further visit in April and then again in mid- June with Bart Godwin to conduct acceptance testing prior to the move back to Australia. The first 50 RIOT terminals were also due to be completed by the end of July 1970 so that the TAB team under the direction of Tom Daniel would have a significant population of terminals to be able to adequately test the software over a 9 month period.

By coincidence, New York had legalised Off-Track betting and released an RFP for an automated system in the early part of 1970 and I was asked to participate in the preparation of the proposal (which was based on the Carbine/Rimfire model) and the presentations in NY. When I arrived in Minneapolis in June I was horrified to find that the RIOT production at the Roseville plant had been "bumped off' by the DART terminal which had been given priority. I was told that I would not have any RIOT terminals available by the end of July - just bad luck!

By chance the evaluation team for the NYOTB system was scheduled to visit VICTAB in early August and after the presentation to this team in NY, I explained to the CDC East Coast Manager (Tom Moore) that CDC would not win the contract because VICTAB would be upset that we had not been able to keep to our implementation schedule for Rimfire. Before the evening was out, he had rung Bill Norris at home and next day Roseville switched off DART and started manufacturing RIOTs. Very impressive, and we managed to have 20 ready for shipping at the end of July and the project was kept on schedule. Needless to say, we did not win NYOTB (but that's another story)!

The initial production of 150 RIOT terminals were to be manufactured in Roseville, with production of the remaining 850 moving to the Cheltenham facility in Melbourne. The project moved to Melbourne in August 1970, where Arnie Hochhalter and John Lestino (CDC) together with Stuart Broad and David Gross finished off the software system and testing commenced in earnest. Phil Stokes joined the team in September and subsequently so did Bob Trafford. The RIMFIRE system went live on May 3rd 1971, spot on schedule exactly 2 years after the contract was signed.

Subsequent Happenings

Peter Dulmanis took over as TAB Account Rep in 1970 and following his move to New Zealand in 1972, Ray Sharp took over the account and negotiated an add-on contract consisting of a sixth 1700 module together with a further 120 D-RIOTS (Dual RIOT - which was the Victorian version of the D-TIM that had been developed for NZTAB) to automate the country areas of Victoria.

Payout fraud where dishonest operators would pocket winning tickets that had been paid out and then cash them again at a distant agency became a serious problem, with losses mounting up.

The solution was to develop a software package to run on the Carbine off line processor, to identify and "match" by ticket detail for duplicate paid tickets - called the Interim Ticket Security System (ITSS):

This called for:
        Original Ticket Sale Site Number.
        Date and Site where winning ticket first paid.
        Date(s) amd Site Number(s) of where the ticket was paid multiple times.
        A reconciliation by Site and 'Pool' of values.
> Earned from 'original ' tickets sold to be paid for the 'pool'.
> Value of multiple tickets paid for the site and pool.
> Value left to be paid for the site and pool, including where total value exceeds original value to be paid.

This system was then used to identify the dishonest operators (who were operating the RIOT that first paid out the ticket) who were then arrested and charged. This had the desired effect to curb this fraud, especially as the agency or branch operators did not know how they were found out.

Another particularly bad problem called the "Phantom Punter" caused tremendous consternation with random amounts being added to collations (and hence losses to TAB). Phil Stokes & Stuart Broad on the software side and Tony Bell & Geoff Hipwell on the hardware side were involved in diagnosing a design problem on the drum.

Phil Stokes has written the following recollection:
"From memory Stuart Broad was shipped back to Melbourne to address the problem. I remember Stuart and I went the College Lawn one night and back at work in a slightly lubricated state where Stuart got the idea for the software to perform every read twice. That exonerated the software and got the focus onto the hardware.

(Tony Bell has writen a separate description of how the hardware fault was found and fixed - see "THE NIGHT WE NAILED THE PHANTOM PUNTER " by Tony Bell February 2008).

There was in fact another Phantom Punter: Des Healy and myself were putting in a hot fix (machine code keyed into the teletype on all live modules to fix some screw up by Carbine meeting scheduling). I got one wrong but we said nothing. Unfortunately Des blabbed to Maurie Henderson when the discrepancy showed. Henderson ripped shreds off me then congratulated me because I have put lots of extra money on a winning Quinella combination. Won the TAB a small fortune that day did Des and I!

One other thing that tickles my memory is Terry Arthur - the practical joker. Remember his pet snake escaping into the false floor? But the joke he pulled on Bob Trafford takes the cake. Bob was very new and I had stuck him on night shift (we were on site live support 7am to midnight?). Every night computer room ops removed the teletype printouts for the day into safe storage. Rimfire's last output for the day (a la John Lestino) was "Rimfire Shutdown good night mate". Terry Arthur typed in "Rimfire Shutdown Get f***ed mate" and called Trafford to complain saying that was unacceptable as there were female operators on shift as well. Trafford was absolutely certain it was my doing. Rimfire program listings were in three very large computer printout folders of assembler code. When I arrived next morning an angry Trafford was still looking at the assembler code."
Marcel Dayan - 10 February 2008

THE NIGHT WE NAILED THE PHANTOM PUNTER

By Tony Bell, February 2008

Marcel Dayan has mentioned the "Phantom Punter" in his history of the VicTAB RIMFIRE Project.   This addendum to his story is an attempt to show some of the challenges we faced, as we broke new ground in the computer systems of the 60's and 70's.

The Phantom Punter manifested itself as random amounts of money being added to the pool, ("wagered"), on random horses or dogs at random times.   The amounts could have been anywhere from a few dollars to hundreds of thousands. It was quite an infrequent occurrence, happening about once a week on average. It could appear a couple of times in a day, but that was exceptional. There were no error displays; the system just accepted that "someone" had placed a huge bet on some rough outsider, (or even the favourite, it didn't seem to care). The operators were the first to notice it; I have a feeling it was Terry Arthur that first asked, (in true Terry Arthur fashion),

"Who's betting $120,000 on that nag? It couldn't run out of sight on a dark night".

Being so random and infrequent made it extremely difficult to diagnose this problem. We looked for patterns - none were immediately apparent. We had our theories; some of them were quite bizarre. For example, it was thought that the computer-room clocks, which worked off pulses from a central "master" clock, were causing electrical interference and somehow messing up the pools.

After months of this, we started to see one constant, (and one only).   It never manifested itself on a metropolitan meeting, i.e. it only ever affected provincial meetings. This was the first clue. Of course, we hardware engineers had to point our fingers at the software. After all, how could the hardware "know the difference" between a metropolitan race meeting and a provincial one? The software guys pointed out that it had something to do with the drum, (hardware), because metropolitan pools were held "in core" whilst provincial pools were stored on the drum. We weren't entirely convinced, but we did start looking very closely at the drums anyway.

The CDC 1751 drum subsystem was quite a peculiar little device.   For those that aren't familiar, a drum was like a hard disk, but with many fixed read/write heads. Early drums were shaped like a cylinder, hence the name. The 1751 was actually disk-shaped, and had one head per track, but many tracks. This made it a lot faster than an ordinary hard disk drive, because the system didn't have to wait for the heads to physically move from one track to another, all you had to do was switch one set of heads off and switch on another set to move between tracks. Switching between tracks was accomplished whilst the heads were over what was called the "dead band". Its capacity was very small by any standard, but it was relatively fast. The whole thing was enclosed in a sealed chamber that was filled with helium gas.   This enabled the heads to "fly" a lot closer to the magnetic surface and hence to store more data.   CDC didn't actually make the "drum" part of the subsystem; it was made by a third party, the name of which escapes me. The interface between the drum and the computer, aka the 1751 "controller", was designed and built by CDC at La Jolla, California.

We ran diagnostic after diagnostic on all the drums in the Rimfire system. Needless to say, all of these tests ran without any error - ever.   This went on for weeks.   In the meantime we pleaded with the software guys to try and replicate the Phantom Punter, but on a more frequent basis. If we could "see" what was going on, we could fix the problem. Finally, after some great efforts by Stuart Broad, Phil Stokes and the rest of the software crew, we were able to see the fault happening often enough to start tracing it. This took a couple of weeks, but we moved closer and closer…..

And the software guys were right, for once.   (……sorry guys, I had to get that in)!   It was, in fact, a drum problem!

Having reached the point where we had gathered enough information to really start scoping the problem, we mounted a concerted effort to get this monkey off our backs.   Ably assisted by our Engineering Branch Manager, Wilson MacMillan, Geoff Hipwell and I took over Rimfire module #6 one Saturday night after the last race.   Wilson supplied the beer, pizzas and encouragement, and we set to work.

To cut a long story short, we traced the fault to a design error.   It had to be a design error - it affected every system to a lesser or greater extent.

The 1751 drum controller contained a dual-purpose read/write "shift register" which interfaced with the drum's heads. When the heads were reading data bits from the drum surface the Most Significant Bit, (MSB"), would arrive at the first flip-flop in the register. Then it would "shift" sideways to the next flip-flop in the register as the Second Most Significant Bit arrived at the first flip-flop, and so on until a complete 16-bit "word" was read off the drum and formed in the register. This "word" was then transferred "in parallel" to the computer memory. The register would then be "cleared", i.e. set to all zeros, in preparation for the next read-shift-read operation. All this would happen in a matter of milliseconds.

In a write operation, the opposite would happen. A "word" would arrive in parallel from the computer and was fed straight into the shift register, with the MSB being fed straight into the top flip-flop of the register and so on. Then the word would be fed in "serial" fashion to the selected heads and written onto the drum surface. Then the register would be reset to all zeros in preparation for the next operation.

So what was the problem? We found that the shift register wasn't being reset to zero on every occasion. If it was necessary to change from a read to a write operation at exactly the same time as we were switching between tracks, the controller would forget to reset the shift register, and the result was a 16-bit INCLUSIVE OR of the last word read off the drum with the first word to be written.   This explained the infrequency of the problem; the chances of switching from a read to a write operation at exactly the same time as the "dead-band" was passing under the heads represented some pretty long odds. It also explained why the bug never decreased the amount of the pool. For those that never studied or have forgotten their Boolean algebra, the result of an Inclusive OR between, say, binary 1001, (nine in decimal terms), with 0110, (six decimal), is binary 1111, (or fifteen decimal). Another example might be 1000, (eight), with 1001, (nine). The result isn't seventeen, it's still 1001, (nine).

Anyway, we added one two-inch wire to the backplane to make sure the shift register was always cleared down to zeroes at the end of each and every read or write operation, regardless of the dead-band period.   Et voila! Mort de la "Phantom Punter"! We couldn't believe it at first. Stuart and Phil's test stopped reporting the error; the standard diagnostics ran without error; Rimfire ran without error. Everything was great.   Adding the wire took about two minutes, and this included the time taken to go and get a wire out of stock!   Just goes to show, as I always said, hardware bugs are easy to fix when you know where to look.

Wilson, Geoff and I wended our various ways home that Sunday morning as the sun rose across St Kilda Road, tired but very happy little hardware folks.   The people at La Jolla took something like six months to publish an official Field Change Order, ("FCO").   They even used my exact words in the explanation of why the change was necessary, but they never acknowledged that some baggy-arsed Customer Engineers from Melbourne, Australia could come up with a fix to their crappy design.

C'est la vie!