I have heard it said that market research has low priority in processing. When I joined McNair Surveys in 1977 (later to become AGB McNair, and finally part of Nielsen), the data processing was done by
our parent company in Australia, and our priority in New Zealand was somewhere after Tasmania.
I was assigned the task of taking up the processing in New Zealand, and was introduced to the business realities and undesirable features of punch cards.
After getting to know the Survey Analysis Program (SAP) at the IBM Bureau and having a few projects under the belt, it came time to process the National Readership Survey.
This comprised a sample of 5,000 and each questionnaire comprised five cards - so that was 25,000 cards plus about 1,000 cards for the analysis program. There were 2,000 cards in a box, totalling 13 boxes of cards.
Data entry girls
We employed several data punch bureaus to do all the data punching. The data entry girls hated punching the questionnaires because they were so slow to punch. Worse still, they had to punch them twice in order to verify them.
As DP manager I spent a great deal of time driving around the city, carrying questionnaires and boxes of cards. One day I dropped one of the boxes, spilling the cards all over the floor. It was when I had to manually sort them all, that I discovered the first undesirable feature of punch cards!
Now, the punch cards comprised 80 columns with a character of data, letter or digit in each column. Each number had its space in the column, and letters were a combination of two punches.
But we know one of the quirks of survey data is multiple response questions. IBM had come up with a clever way of dealing with these on the punch cards, courtesy of multi-punching. The format was known as column binary and it meant you could have multiple digits punched into one column.
That was great until I discovered the second undesirable feature. It was possible for a respondent to answer ‘yes’ for every response option. In that case all the numbers in the column were punched into holes. It was as if those particular cards were perforated, so they could easily tear apart!
One run per day
Finally all 26,000 cards were punched, so I took them to the bureau for processing. We were assigned one run a day. But the program would take several attempts to compile and therefore several runs were going to be needed. The operators soon got sick of repeatedly feeding 26,000 cards through the hopper.
When I noticed a few of the data cards coming up as ‘invalid’, I would have to duplicate them onto new cards. And as I fixed them for the next run, there would be more that were invalid the next time. I had discovered the next undesirable feature of punch cards, which was that the card reader would wear them out!
When the operators asked in exasperation how many more times they were going to have to feed in the 26,000 cards, I told them of the predicament. In the end, the solution proposed by one of the boffins was to put the data onto a tape and delete the invalid records. Luckily there weren’t too many.
In due course I decided there had to be a better way. Indeed, I loved the analysis part but was not that excited about the operational mechanics of collecting or processing data. So in part, punch cards helped inspire me to start a market research technology company and get value from data in a better way.
This blog was previously published in the Market Research Society New Zealand's (MRSNZ) InterVIEW, July 2012.