I said I would post something about my experiences of non-sharing of data and then forgot to do it. So here is a little story.
For 3 years or so I worked on the British Regional Heart Study (BRHS) as a clerical assistant. I coded quite a few bits of the questionnaires, including the social class codes (as I had 2 degrees in sociology this seemed to them like a logical thing to get me to do, which it was really). Eventually I decided I ought to go and try to get a PhD (long story, some other time). I had gotten really interested in epidemiology, could see a lot of scope for a medical sociologist to work in the area, and needed a 'credential'. As it happened, the BRHS had surveyed 300 men in each of 24 British towns in 1978-9, just before the economic crisis caused by Thatcher's policies. Several of these were heavily industrial towns, whose populations had experienced massive rises in unemployment. It was a natural experiment.
So off I went to Edinburgh university to study with Adrian Sinfield, as I wanted to study unemployment and health and Adrian was a world authority on unemployment. I had a study design in my head, based on one town, Dunfermline, where the BRHS had studied a sample of middle aged men, taking lots of biomarker measures as well as sketchy social information. My plan was to team up with experts from the Edinburgh University school of nursing, which was part of the social policy department (where Adrian was Professor). I would design a life event questionnaire and I hoped that a nursing Masters or PhD student (even in those days -- early 1980s -- nursing was a degree subject in Edinburgh). "we're not doing your bloody fieldwork for you!" was the response of the nurses, quite rightly. So I said OK you teach me to measure blood pressure and lung function and I will teach you about labour markets and life events and we can do it all together.
The next step was to get hold of the data for Dunfermline, which was in part a mining town that had experienced massive economic change, Adrian seemed rather dubious about this so we approached Margot Jefferys, my mentor when I did the MSc in Medical Sociology, who knew the Director of the BRHS, Gerry Shaper. Her response was not encouraging "You might try extreme flattery" she advised. In my innocence, I had absolutely no idea why everyone was acting so strangely about this. After all, I had worked on these data already. In fact in Dunfermline I even did the interviewing to fill in for one of the nurse fieldworkers who had broken a finger. So I had personally met most of the participants in the study in that town.
No way, said Shaper, the GP practice from which we drew our sample would not tolerate someone going back to their patients. I said, of course it would be up to the GPs to ask permission from their patients for me to get in touch with a small sub-sample. I would not be appearing out of the blue. Based on other experiences, I was pretty sure that a lot of people would be only too happy to talk about what had happened to their jobs. And the biological measures I planned to do were non-invasive, unlike some of the ones people had agreed to in the main study. I had wanted to take a much more qualitative approach and devote more time to listening to people's stories of the recession and their experiences. We already knew from the original study and some linked data from GP records and the NHS central statistics organisation that men who were unemployed were at greater risk of health problems. But what about those who later lost their jobs as a result of the 1980s recession? To this day there has never been such a perfect opportunity to do a study of this kind.
I consider that protection of the privacy of research participants is of the very highest importance. Anyone who reveals personal information about someone who has consented to be in a study should be banned from human research for life. But 30 years after all this, I know much better now the real reasons for reluctance to share research data and it has nothing to do with protection of participants. I am reminded of a much more recent experience where a teacher obtained a small subset of study data (not BRHS) to use as an example for students. Once the course was finished, the study director demanded she give the data back and destroy her own copy, When I asked why, the director responded: "What if one of the student discovers something we have never seen?" This is a shameful attitude, but much closer to the real reason for reluctance to share data than crocodile tears about confidentiality.