Learning from Corporate Memory or Using Myths to Support the Current Ideology
At the 4th International Driverless Vehicle Summit in Sydney at the end of October 2019 a representative from the NSW road authority mentioned the hugely successful Sydney Coordinated Adaptive Traffic System (SCATS) that, he said, had a history spaning the last 20 years or so.
The reality is that SCATS has a history of over 50 years. While not blaming the presenter, it does suggest a lack of corporate memory and awareness about the real impacts of some of the monumental developments in its history. This is important not just to feel good about the past, but to ensure we can continue to learn from our previous experiences.
Now is an opportune time to reflect on the development of SCATS as the NSW government is outsourcing the maintenance and development of the system. (Read more: https://tenders.nsw.gov.au/rms/?event=public.rft.show&RFTUUID=9F57448F-00A7-F153-8A2D3A3FA2A1BFAA )
It is essential that the conditions under which this is outsourced must represent the broad goals of why the system was developed in the first place.
History must be more than what you get at an awards ceremony: a summary of a few technical facts and lauding some of the people involved. These are valuable but unless we look at the life and times of the project, we will only use history to justify the current ideology.
In the case of SCATS, it was a world-leading system to improve the efficiency of traffic movement. It was so successful it was sold to countries around the world. But the lessons of SCATS are much more than just “We can and must develop new technologies”.
The lessons involve recognising the destructive nature of bad government department structures; the difficulty in recognising and dealing with very talented technical people; the need for a reasonable number of managers to actually understand what the technology is trying to do; the limitations in negotiating from your own narrow perspective; legitimate concerns in introducing new technology when some of the existing technology is still very good.
How often have we heard technical people lament that solutions are stymied by the political environment or poor management? We all know that this happens. We need to understand history so that we don’t just complain but we review and learn what did and didn’t work.
This is by no means a complete history of SCATS but just some reflections, identified through conversations with some key players.
Life and times of SCATS
In 1974 the Sydney Area Transport Study (SATS) was published.
The major objective of SATS was to show what had to be built to serve the transportation needs of the Sydney region through to the year 2000. The plan incorporated an extensive list of huge projects such as multilane freeways and new and improved rail links throughout the metropolitan area. It was totally unrealistic and Ken Dobinson, who became a director of the DMR, has said that Sydney could not have built all these projects in 200 years let alone 20 years!
The prevailing culture was that we needed to build our way out of congestion. Making the existing more efficient, played second fiddle to grand projects, at least in the political and public debate.
As if to emphasise the unrealistic nature of the SATS report, it was also the time when road and rail budgets were low.
Limited funds meant that traffic management was a “spot fire” approach – you picked the worst congestion spots and tried to solve the problem.
Yet there was a group of people, initially in the Department of Motor Transport (DMT) and then into the Department of Main Roads (DMR), who were diligently working on traffic management, how to get the most out of the existing system in as safe a manner as possible. Not just looking at “fire fighting” but at large, complex ways to bring more efficiency.
What were the key elements that made SCATS possible?
To achieve SCATS there had to be a range of personnel from those who could envisage a new direction, those who could develop specialist areas required, those who understood computers and those who understood traffic. You needed managers who could appreciate and facilitate the technical excellence that was the core part of the exercise and managers who could manage up to their bosses to convince them to commit to the project.
The father of the technical SCATS system was Arthur Sims. He came to the DMT from IBM. The DMT Chief Engineer Rae French hunted him after being greatly impressed by his work in developing the first stage of the inner-city coordinated system (eight sites commissioned in 1965).
Arthur did not have a tertiary qualification but when asked by the Singapore Government what his qualifications were, Ken Dobinson wrote back simply “A genius”.
There were of course many others who contributed but any list would never be complete. I will mention a few in passing with immediate apologies to those we have not. Names pop up in conversation from many people’s experience such as our interview for this newsletter with Andrew Leedham who was part of taking on SCATS in Adelaide and who mentioned his appreciation of, among others, Alan Finlay (who in semi-retirement volunteers as the sub editor of this newsletter). More names will come up in future.
Apart from geniuses and computing power, a critical piece of technology was the loop detectors that could be installed at traffic lights to sense the presence of cars. Previously there had just been road strips that could only note that a vehicle had been there, not that it continued to occupy or be passing over a space at varying speeds.
Loop detectors are not perfect (Frank Hulscher and others made further development with them so that they could be used for a input to an adaptive system). It proved impossible to develop algorithms to adjust the traffic signals based on when cars were detected by the loops. Peter Lowrie then had a brilliant left-field idea of measuring the gaps between the vehicles, when cars were not there! It proved a winner.
How good and how important was SCATS?
Although developed in one city, in one state, in a small (population) country, SCATS has been an international success. It has now been sold to some 40 countries, in over 1800 cities and operating in some 42,000 intersections. It became the benchmark in embracing new technologies to be used to improve efficiency in traffic flows in urban areas.
Some important aspects of the system are:
• It made signal timing changes in real time based on the actual road volumes and densities
• It provided extensive traffic data in a manner not previously achievable.
• It was a transport system that embraced area wide applications.
• It developed a system where an experienced controller could sit in a helicopter, adjust the signal parameters, and immediately observe the impact.
• Because of its wide area application and its ability to adjust to specific land use developments, SCATS added further opportunity for the roads authority to become involved in planning issues.
At least one weakness was that SCATS was never incorporated into all the latest technical management applications such as lane monitoring and other ‘smart’ freeway management systems.
Structural and Cultural limitations
Not all the strengths and weaknesses in the development and acceptance processes were technical. Cultural attitudes, Government structures, internal and external politics, and management errors were all part of the mix.
- Government departments in NSW were established by transport mode (Road and rail being the big players)
- This very technical area that was pioneering new ground was difficult for some managers to understand. Ken Dobinson spent hours with Arthur Sims getting a good understanding of the system. Ken was never going to program a SCATS computer, but he knew what the system had been built to do.
- There are several stories about times when the technical people agreed not to tell the senior management about certain developments as it did not suit the politics of the organisation. When you hear people talk about its history and they use expressions such as “the rumour was……..” you sense how much management can help or hinder the process.
- Countries had their own (mostly fixed time) systems and many, especially the United States, struggled to accept that a minion like Sydney in Australia could develop something better than themselves.
Further development and maintenance
No system can stand still if it is in a competitive world.
SCATS 2 was a major project undertaken to try and bring fresh thinking into the system. Consultants were brought in and given a clean sheet of paper to develop an improved system. Those who had worked on and were still maintaining the original system, were specifically told not to communicate with the new team.
SCATS 2 was a failure.
SCATS is not just high-powered computer and twiddling a few broad parameters now and then. IT needs resources to ‘fine tune’ operations at the tactical and local level. It needs on-the-ground understanding right down to the nuances of individual intersections.
There is a strong current of feeling that there is insufficient understanding, expertise and commitment in the government organisation to maximise the benefit of the system. One person feels that it is running about 20% below its capability.
My own limited personal reflection is that some intersections in particular look like they are operating in a peak period cycle at non-peak times.
Perhaps it is not surprising then that government wishes to outsource SCATS’ maintenance and development in what is understood to be a partnership arrangement. This is not the first time it has been suggested.
A few facts and experiences that emphasise the special nature and commitment of those who developed SCATS
Genius with a golf club
Ken Dobinson says: “To do the mathematics to the algorithm to release the time between signals Arthur’s work was sent to a government section for their analysis. They eventually came back with a 60-page report. Ken was up to about page 20 reading it, when Arthur stormed in and said they had got it all wrong. Arthur’s solution was to go to the golf course and “belt the hell” out of some golf balls. Ken says Arthur came back the next day with one-page on which was a graph and an equation. After a further three months of checking within government, Arthur was proved to be right.
Dedication beyond the call of duty
The original system was developed for the best hardware at the time: PDP computers. As time went on this hardware did not have the capacity to handle the developing system.
Ken McCallum had accumulated too many holidays and he had to take 6 weeks leave. In that time, he and Graham Davis re-wrote the whole program so that it could be operated under a Windows system on a PC. It was an act of total dedication to the cause.
Arthur nearly left the project
Harry Camkin tells the story: “I suspect that very few people know that we very nearly lost Arthur to our cause about 1970, when he was so disenchanted about being moved from the Traffic Control Centre to the Traffic Accident Research Unit to help Rae French computerise its accident records. When I alerted the then Commissioner for Motor Transport, David Coleman, of Arthur’s plans to resign, he, being similarly impressed by Art’s capabilities, told me in no uncertain terms to “just fix it!” We cooked up a unique contract arrangement with some very non-public-service perks for Arthur and Graham Davis, and the rest as you know is now history.”
Constructive Relationships
The expansion of the number of traffic signals in the 1960s was having an impact.
Sydney also developed a traffic control centre with some television surveillance and an ability to adjust the signal timing from a location not at the site.
It had been law that only the Police could turn off traffic lights so the police had to be present in the control centre. It was a time when the interaction with the police force, including at the AITPM, was strong.
People like Jim Giffin, who was a technician who ran training sessions in signal operations, developed a constructive interaction with the NSW Police.
His father would have been proud
Arthur’s father was an iron moulder and core maker. Arthur said “he used to work at the Eveleigh workshops near Redfern and he used to make all the cylinders for the old steam trains. The 36 [class] were one of his pets - the ones that used to be the fastest mountain train in the world and used to go up the Blue Mountains.”
The Eveleigh workshops have now been turned into a technology park and one of the rooms is named “The Arthur Sims Room”. His father would have been proud.
Sadly Arthur has passed away but you can hear him tell some of his story in a short AITPM news feature: “Why academic qualifications aren’t everything”.
What of the Future?
How should we remember how SCATS was developed and what should we look for when its maintenance and development is outsourced?
One comment from Alan Finlay is: “My worry is that SCATS will become more about ‘big data’ that can be sold, rather than rigorous traffic signals engineering. From my observations there is a dwindling number of skilled people who really understand the intricacies of the system.”
Harry Camkin has a few thoughts that are well worth repeating:
“The biggest risk as I see it is that of restricting the “feedback” as to operational changes indicated from practice, or strategic/tactical changes in objectives to address changes in transport planning goals emanating from new government policy directions. This is not to forget benefits that might be achieved from extension of the technology to facilitate other community objectives.
“These risks appear to run counter, in large measure, to the potential benefits in tightening the link between transport policy and transport “customers” that the consolidation of agencies under TfNSW offers.
“We would certainly not want to see the financial objectives of commercialisation overwhelm the technical ones, any more than we would like to see the departmental skills base and analytical capability further eroded.
“I think we have often, to little avail unfortunately, canvassed the extent to which “outsourcing” has denuded very capable, if sometimes a little misguided, public sector agencies of the capacity not only to plan, but also to evaluate and monitor public services that have been privatised. That is not to mention the twin proclivities in the absence of corporate memory, to fail to learn the lessons of the past and to re-invent the wheel.