Recordkeeping Roundtable

In part 2 of our chat with Ellen Broad, we talks about privacy and changing attitudes to data about – and of – ourselves.  Ellen’s book, Made by Humans: The AI Condition is available from Melbourne University Press, or from all the big online book retailers. You can read Ellen’s bio after the transcript.

https://rkroundtable.org/wp-content/uploads/2018/10/episode-5-ellen-broad-part-2.mp3

Transcript

CF: But I mean data is to those companies, whether it’s a specifically focused company that does a very particular thing or whether it’s Google is it’s gold. I’m living in San Francisco now, and we’re experiencing a gold rush bigger than 1849 because data that is collected and amassed and used by the big tech companies here is it’s just the gold of the 21st century. And so the stakes do get very high and it gets much harder for the public authorities to assert that control against these tech giants. Do you see, I mean, and this is also a bit of a side issue I suspect, but the sort of centralization of the few sort of massive companies that seem to be gobbling up all of our data, do you see any chance of a change away from that kind of semi-monopoly that they have?

EB: So I find the language like data is gold and oil and their comparisons to precious metals or finite resources really problematic because it shapes even how those companies treat the data that they have. So they sit on it as having innate value, so you know you accumulate. When I think of gold, I’m like they’re accumulating massive reserves of what I think of as inherently valuable information that can be exchanged for money when actually data is also incredibly fragile and difficult to preserve and very quickly out of date and cannot really just be kept in a database and sat on and still have that same value preserved.

It doesn’t innately grow in value over time. If anything, if you fail to think of it the way archivists think of information, the quicker the value in your data will decline as in, even the way that organizations approach data collection and storage, I don’t know many technology companies that think of preservation of the data that they have or the need to introduce quality controls around data collection itself to ensure a reliable historic record. I don’t know many organizations that actually approach data with that longevity in mind. It’s just, let’s pull it out of the ground to use the gold language. Let’s rip it out and see as much to them.

And then unfortunately if they just focus on, “Let’s get as much as we can and store as much as we can,” they’ll open it up eventually and realize that a lot of the value might have crumbled away. And that’s where I think where we have a reckoning coming, as in we have just kind of gotten into this mindset of we’re in a gold rush. The more you have, the more valuable it is, but we’re not investing in curation, preservation, ongoing access, long-term access…

CF: Yes. And understanding where… What I’ve seen certainly in some of the settings that I’ve had a bit of exposure to hear that a lack of kind of being able to look across a number of areas of business and understand the way the criticality and the value is and where to focus, if it’s on preserving that historical record, and all data is not created equal. And so what I’m doing in my current work is trying to kind of map out some of those, I guess, understandings of business to overlay on the technologies that are collecting, storing, sharing, destroying data.

And it’s fascinating stuff, but I wasn’t here to talk about me. I have a third question which is going to get to the privacy piece ’cause it is so much a part of this, and anyone who has a heartbeat and goes on to the internet understands that we are every day making choices about what we are willing to give away in return for a little bit of information about us or sometimes a lot of information about us in terms of convenience and you touched on it early when you sort of were talking about how we could be given certain preferences based on what we’ve done in the past and all of that.

So, I feel like though in the last twelve months globally, there’s been this kind of really interesting recognition of that bargain or that exchange. And with the introduction of things like the general data protection regulation in the EU, there’s a new law coming in in California, which has a lot of similarities to that, and at the same time, a lot of high profile cases of companies in particular, not just companies, governments as well, but companies like Facebook, breaching their customers’ trust by sharing data with third parties and in that case, Cambridge Analytica. So I guess the question I have is, where do you see this all heading? I mean, we’ve got the regulation on one hand, we’ve got data breaches seemingly happening at an ever increasing pace. What’s your kind of crystal ball prediction on where this is all going?

EB: Maybe I can talk about where I hope it’s going.

CF: Okay.

EB: Where I’m afraid it might go in part is I think like it’s really hard to predict what the world will look like in a few years because at the moment it seems like we’re almost being driven in several directions and it remains to be seen kind of what approach wins out. So definitely something that I think that is happening that is very positive is that we’re seeing a responsibility for safeguarding and making choices about personal data shifting from individuals on to entities collecting that data and that is I think a shift in responsibility that’s reflected in instruments like the general data protection regulation in the European Union, but also a reflection of how well sense of the responsibilities of organizations in this space is changing.

I think in the book, I have a throwaway half page reflection on reading that excellent book, The Immortal Life of Henrietta Lacks, which looked at specifically the story of Henrietta Lacks and her cancer cells, HeLa cells, that have been used widely throughout the world for cancer research, but were taken without her consent and the family’s battle to get control over that tissue back today. But what really struck me just looking back at kind of consent issues around blood and tissue in the 20th century was how much the debate has played out in almost exactly the same way with the same language.

We, in privacy and they went through exactly the same mindset shift in the medical sector was it started out being, this is just the way the sector works, if you don’t like what we’re doing, it is up to you to stop it from happening. That’s really interesting if you look back, you know, the language that Mark Zuckerberg uses and a lot of technology founders. It’s, if you are unhappy with the way that your data is being processed, don’t use our services. And how could this be unethical if this is just the way everybody is? And this is the way everybody uses these services.

And just as we saw with kind of consent issues and autonomy concerns in relation to blood and tissue, I think that’s what we’re starting to move into with personal data is actually that it is not quite often individuals don’t have the power to shape how information about them is collected, they don’t have any meaningful mechanisms to control it. And that even if you’re not in these spaces, information about you is being processed anyway. So number one, I think that has been a really good and positive shift that we’re seeing responsibility move away from individuals more on to organizations collecting data, and that should have flown effects in how we treat data breaches, in how we consider ancillary services, secondary services, provided using data that’s being collected.

So I think that is positive and I hope that that continues and kind of becomes the norm in other jurisdictions. What I do see as well that worries me because it could shape the way that we think about information in the future is the increasing approach to personal data as a commodity that should be bought and sold. So, as well as a sense that if you’re an organization collecting personal information, you need to follow certain practices and be responsible. We’re also seeing a lot of discussion around what is personal data worth. This is data that I own, how can I monetize it? Here are some services that can help you monetize your data. And information is not a commodity like your house or your car. Your providing information to somebody else doesn’t prevent you from having access to it yourself.

It is something more than, in my mind, an asset that can be bought and sold, particularly because we have relied on information about people throughout history to help us make lots of different kinds of decisions. So if we move so far down the path of ownership over personal data like individual ownership and control, I think it’s really hard to divide what is mine and what is yours, in relation to information. Like you and I, who owns this recording? I mean, I know we’re under copyright law, which of us owns it? But as information where I’m revealing information about myself and you’re revealing information about yourself, control over what happens with this is something that’s hard to just separate. It’s hard to just cut in half and give me certain rights and you certain rights. I have mentioned other people in our conversation. It’s just networks are complicated.

And I think we just can’t forget that, that we live in these complex webs of connectedness to other people and if we try and lock down information too much, we might prevent some really powerful uses of information as well.

CF: Yeah, I mean it’s… What you’re saying is really resonating with me. And that, of course, archival institutions have been about capturing data at scale that represents those networks and how they function. And of course, Mark has a full of personal information that has been collected and also makes me think about the interesting debates that have happened in more recent years around the census collection and around people… Now, you have people wanting to walk out on having themselves recorded as part of the census partly because they mistrusted how the agency charged with doing the census would store and handle that data, which is probably not an unreasonable fear, but also I think part of it is that sort of distancing themselves from wanting to be part of a giant data collection effort. But then on the flip side, that giant data collection, if it is there to help build roads and schools and is part of being part of a community, part of a society. I’m not sure if you touched on the census issue. I can’t remember. In the book, did you have a look at that?

EB: I didn’t touch on it. I removed it. I remember writing about it early on, but it was more because just I couldn’t pack in any now into the book but the census is a really interesting example of an instrument as you say that’s collected for really socially beneficial purposes and the desire to link census instruments across kind of four-year period is not malevolent. They’re not trying to create a surveillance economy in which every citizen can be monitored but at an aggregated level, that kind of societal level could help social researchers, statisticians, policy makers kind of understand trends that are changing in society.

The complexity though is that because we collect so much information digitally now like I was incredibly sympathetic to the concerns around linkage in the census and digital collection of census data. Because this information is digital, it is inherently easier to share and easier to access. And so while everything being on paper created a kind of friction, it disincentivizes easy linkage and make distribution of information to lots of different actors because it’s just expensive and difficult.

And I say this, I’d probably be preaching to the converted given that this is an archival audience, but I think about this a lot in the context of a family history research. My family is not a famous family, it’s a very ordinary family background. And then when you find these tiny glimpses of past ancestors in official records held in state local national archives, there is a real thrill.

Because they are the tangible manifestations of people that came before you and are related to you and you wanna have access to that and that is digital now means that I can search for migration records for my great grandparents that are in a West Australian archive while I’m in Canberra. So I was seeing this real benefit there, but I also have friends who have notorious families who hate the fact that their family is instantly searchable and that information is much easier to access because it’s been digitized and is accessible. And so, I mean I think that’s just the complexity of it, and I don’t think that anything should be changed, but it just to me, speaks to me how we can’t forget the importance of preserving this kind of identifiable individual information because for every person that it might be a problem too, there is someone else that benefits by its presence, by its existence.

CF: Yeah. Look, yeah, I think there’s a lot in what you’re saying and I think one of our challenges as recordkeeping professionals and archivists is navigating this transitional period and coming to groups with what being born digital or digitized means in terms of access, in terms of availability, in terms of impact on those people living in society because of course time periods are getting much more squished down as well. We’re not waiting for decades until things become available. So Tim Sherratt has written some brilliant stuff about this and about different concepts of access when you’re thinking in terms of digital. Yes, he is. So I’d encourage people to go and check his work out. But look, I probably need to wrap it up. I think that was a really nice observation on which to end actually, and it called to arms a little bit for recordkeeping people, but I guess all that I wanted to say to finish up is do go and read Ellen’s fabulous book, Made By Humans: The AI Condition. And thank you so much, Ellen, for making the time to chat with me today.

EB: Thank you so much, Cassie. I really enjoyed it.

About Ellen Broad

Ellen is Head of Technical Delivery, Consumer Data Standards for CSIRO’s Data61. She returned to Australia from the UK in late 2016, where she was Head of Policy for the Open Data Institute (ODI), an international non-profit founded by Sir Tim Berners-Lee and Sir Nigel Shadbolt. While in the UK Ellen became an expert adviser to senior UK government minister Elisabeth Truss on data. She has also held roles as Manager of Digital Policy and Projects for the International Federation of Library Associations and Institutions (Netherlands) and Executive Officer for the Australian Digital Alliance.

Ellen’s written and spoken about AI, open data and data sharing issues in places like the New Scientist and the Guardian, for ABC Radio National’s ‘Big Ideas’ and ‘Future Tense’ Programmes and at SXSW. Unrelated to her job, Ellen built an open data board game, Datopolis together with ODI CEO Jeni Tennison, which is being played in 19 countries. 

Recordkeeping Roundtable is made with Jetpack Podcast. Full show notes and every episode at https://rkroundtable.org/

In this latest episode of Recordkeeping Roundcasts, we talk to Ellen Broad, author of Made by Humans: The AI Condition, about the way that rapidly advancing technologies like artificial intelligence and machine learning are being deployed in business, government and society, and the wide ranging implications of their adoption. Ellen and the Roundtable’s Cassie Findlay discuss on real world results flowing from machine decision making, accountability for the use of these systems, the role of recordkeeping and archives, and changing perceptions of  privacy in the data economy.

Made by Humans is available from Melbourne University Press, or from all the big online book retailers. You can read Ellen’s bio after the transcript.

Transcript

CF: I am thrilled to be talking to Ellen Broad today. Ellen is the author of Made By Humans: The AI Condition. There’s a link available from the site too. There’s information about the book and where to buy it. I heartily recommend it. It’s very timely. Look at the ethics and the societal implications really of what seems like ever increasing gallop towards the use of artificial intelligence and machine learning in so many different aspects of our lives. And of course, for the recordkeeping people like myself and hopefully people who follow this series of recordings, there are lots of things about the adoption of this technology that make us think about what our jobs are and potentially what our jobs might be in the future as far as keeping evidence in the way that these systems are set up through to trying to promote accountability around decisions to deploy the technology. So, first of all, hello Ellen, and thank you very much for joining me.

EB: Thank you very much for having me.

CF: So I’ve got three questions. And I think the first one is probably an opportunity for you to help our listeners just understand what it is we’re talking about a little bit because not all of us come at this with lots of background knowledge. And so, the first of my three questions is, in your book, you talk about how we seem to be moving from lower stakes AI systems to higher stakes AI systems. I was just wondering if you could explain a little bit more about what you mean by that and give us some examples of those higher stakes systems.

EB: Sure. So perhaps it’s worth walking it back a little bit to talk about the context in which I refer to lower stakes and higher stakes systems because the book isn’t about artificial intelligence in general because there are many, many different technologies that seat under that umbrella, there’s virtual reality, robotics, drones, what I’m interested in, in the context of the book, is the increasing use of data to make predictions and decisions about people. And we have started doing this more and more in our day-to-day lives. So the lower stakes systems that I talk about are systems helping us choose what we watch next on Netflix for example, or the information Google uses to shape what it is that we see in search results.

These are in some context, even those examples, are not what you would call lower stakes. There’s a lot of discussion, increasing discussion, around how Google shapes the information that we see in the world, for example, but we’re moving from these systems that essentially are used to make individual predictions to us about content, what we might like to watch, what we might like to buy, if we’ve just purchased this book, we might be interested in this one over here into starting to use the same techniques to make what I would call higher stakes decisions. These are decisions about the kind of person that we might be. So whether we’re trustworthy, whether we’re a good fit for a job that we’re applying for, whether we should be accepted to university and the types of degrees that we should study.

And the reason I call these higher stakes decisions is they have real world implications. I mean Netflix recommending me a terrible movie and not understanding my preferences doesn’t shape me in the same way that being mischaracterized for a job does so the consequences are more significant and these predictions are just genuinely quite hard to make. Predictions like the jobs that we might be interested in or be suitable for are shaped by a range of factors and our kind of past preferences and experiences are only one part of that. So that’s kind of where I’m coming from in lower stakes systems and why I think we need to focus on those higher stake systems more.

CF: I think I know from reading your book that you did travel around the world, and you talked with a lot of people, and you mentioned a lot of very specific examples, could you mention an example of one of those systems that has been put in place where it is something as serious as a job or a decision or a university application that is in place now?

EB: So one of the really key ones that I talk about in the book that has been getting a lot of focus and has really drawn attention to the complexities in these kinds of systems is the COMPAS sentencing algorithm, which is used to predict who is most likely to be recommitting a crime, and that’s an example of a probabilistic system that’s using hundreds of variables to estimate a person’s likelihood of re-offending but in practice was discovered to skew certain ways in its output. So it overpredicted on African American defendants being likely to recommit a crime and underpredicted on white defendants being likely to recommit a crime.

And the reasons for that are complex and difficult to unpick and solutions to that problem are also complex. And a lot of discussions in machine learning have started to focus on how you quantify those kinds of outputs from a system and how you try to change them or at least articulate why it is that that is the outcome. I talked a little bit about the growing use of predictions in job recruitment, but COMPAS is probably the most scoped out example of a high stake system in the book.

CF: And, I mean, this does come get us sort of fairly rapidly to what is I think for me one of the really key issues in this and so I’m happy to go there quite rapidly, which is around the lack of accountability, the lack of transparency for the algorithms that are making the decisions that have such significant effects. I don’t know who created the COMPAS application, it was a proprietary company?

EB: Previously called Northpointe but I think they’re now called Equivant.

CF: Yeah, and all the sort of complexity that you mentioned that goes into how that algorithm arrives at its decisions and what data is it analyzing, where does it come from, what’s the provenance of it, what biases went into it, and we just don’t know in most cases, I think is fair to say. Do you wanna speak a little bit to that?

EB: Yeah, absolutely and definitely the one thing that I’d preference talking about the opaque nature of a lot of these systems is that sometimes our human decision making is even harder to unpick and has similar effects and all these systems have to learn from those previous kind of human decisions and human patterns. So, in sentencing contexts, for example, judges, parole officers, policy makers have tried to predict who is likely to recommit a crime in a variety of ways for decades, nearly a century, in fact, because it’s a really crucial part of criminal justice and policing is you wanna make sure that you are releasing people quickly and expediently into the community where they serve no further risk while trying to identify and kind of protect the community from genuine risk in your population.

EB: And that’s a very emotional issue. I talked a little bit in the book about that this is a genuinely hard problem to solve, and we’ve been trying to solve it for a long time. And so the Northpointe/Equivant algorithm is really just a kind of a manifestation of the same type of decision making except this time it’s in software. And so while it’s opaque on purpose in certain ways as you say, it was very hard to uncover exactly what variables Northpointe/Equivant used for their prediction algorithm. It was hard to unpick what the training data was, information about testing that had been undertaken, the range of the kind of accuracy that the system had. That’s all proprietary information that’s been quite difficult to extract over time. Some of it is increasingly being extracted, but it was essentially black box on purpose, but because it is software we can actually at least quantify and identify the sources or the reasons for certain kinds of bias and decision making that in humans we have a feel for but can’t quite systematically unpick.

So COMPAS, what was really interesting about the COMPAS algorithm was not only that, yes, it’s an example of a proprietary algorithm that is being designed to have real world impact, it makes decisions or it helps I should say, helps judges and parole officers make decisions, so they’re relying on it to help them make other decisions, and it’s difficult to scrutinize. But what I found also particularly interesting about the Northpointe example is that follow-up research has indicated that the complexity of the algorithms, so they promote it as having these hundreds of variables to make its predictions and some follow-up research that I say in the book has indicated that they can achieve exactly or close to the same levels of accuracy using just five or six variables about individuals that adding all of these extra data points doesn’t necessarily improve the quality of your system. And I think that’s a crucial kind of point to keep coming back to particularly as we talk about big data is that these more complex algorithms, these algorithms that use lots and lots of inputs are may not be any better than a survey with five questions.

CF: That’s so interesting. And, yeah, I think that when we look at your earlier point around that humans have been making these decisions with all of the information that they have at their disposal and with all of the sort of potential for error that comes with it, brings to mind the way that we kind of scrutinize and try to validate that decision making in the old world, which is through things like FOI, examining the evidence behind decision making but with the increasing adoption of, you know, the increasing partnerships with the private sector in government, some of those opportunities to scrutinize the data, even if it was back in the day it was a sort of analog data, but now are being cut off. Have you sort of seen any evidence of better practices in government about trying to force accountability in partners where they’re performing a public service?

EB: So specifically in relation to predictive systems and kind of AI in Australia, not so much, but in the US, there’s some really interesting developments emerging around regulating automated services provided by private companies and partnership with government. I’ve been watching New York cities algorithmic task force with interest. They are looking at how they might hold the providers of automated services for New York City accountable and what that might include. What I think we can look at is like we use a variety of mechanisms and structures, rules, laws, principles, expectations to shape the way that private sector companies partner with government to deliver services in other areas. If you are providing medical tools for example or medical services on behalf of a government provider, you have to comply with a range of rules that shape how you deliver that service.

We have rules around how employees acting as government public servants utter, act, and operate. It’s never been a kind of free for all. It’s never been unfettered. There’s never been absolute freedom for private sector partners to government. And I found the discussion of IP in relation to automated systems kind of a red herring. We have never… The origin of intellectual property law has never been absolute. It has always been a balance between information that needs to be accessible in the public interest and information that should be protected and we manage that balance in every sector using both explicit exceptions on the intellectual property laws and rules that are just supposed to shape responsible conduct in that sector.

Pharmaceutical companies, for example, can’t keep absolutely secret everything they do in relation to the pharmaceuticals that they offer to the public because they have a responsibility to protect the public from harm and part of that is exposing information about their products undergoing testing, being audited, having to demonstrate accountability, so I find the emphasis on IP in relation to technology a total red herring because IP has never been intended to prevent information being accessible because it benefits the public and helps us make sure that people are protected from harm and able to make informed choices about what’s going on around them.

CF: Yeah. And I mean, I know from my own background working in government recordkeeping that the sort of negotiations and arrangements that go into government agency outsourcing a public function of some sort would touch on recordkeeping. But even as recently as sort of five years ago when I was still working in government, it was very sort of old-fashioned type of, oh at the end of the contract, you will give us all your records sort of thing rather than building in mechanisms for sharing of real-time information or at least expectations for a greater transparency where there were programs that were directly affecting government by proxy decision making. So I suppose what you’re saying is that there needs to be a kind of some more levers, some more mechanisms, start to become normal in the way that government strikes these agreements?

EB: Yeah, I would classify our procurement in relation to software, which is this is all software. When I talk about artificial intelligence and predictive systems, they’re all still software products being developed, and I think we’re still pretty immature in how we procure software and that’s partly because there’s a relatively lower understanding of kind of how software works and what you need to know and understand if you’re procuring software in order to make sure that you can understand how it works and build on it and get someone else to fix it. I used to do a lot of work in the UK around procurement of data. And quite often one of the real challenges that we would run into is kind of in a similar way when you say a lot of the requirements in procurement in relation to information is give us your records at the end and similarly in the kind of datasets, there might just be a clause saying you should, you must give over all of the data that you collect at the end, but they wouldn’t specify.

Sometimes you would see language like you must give over and anonymize the dataset at the end of the project, but actually what is really useful for an organization to have are the methodologies that went into collecting data. So if you wanna repeat the process, it’s easily repeatable so that you can understand how data was collected. You want to know where it was collected from, who it was collected from in case you need to go back and recreate it. You don’t just want the end result typically. And even getting the end result may not even give you the necessary rights you need to be able to do whatever you want with that anyway. So it’s just I think we’re still figuring out exactly what it is that we need in procurement.

CF: Yeah, and you make the point in the book I think around understanding the mechanisms and the methodologies behind the data collection from the point of view of fairness and the point of view of the kind of silences in the data that go back to the way archives have had silences around particular groups in society, particularly marginalized groups, and so sort of having some way to have a greater involvement as the public body that has hopefully society’s interest at heart in sort of engaging with those collection methods upfront is a much more appropriate way to go, I would have thought.

EB: I remember I actually think you had John (Sheridan) share it in on your podcast before. I haven’t really judged it from…

CF: Yes.

EB: Yes, but he and I always have fascinating mindblowing conversations about data in archives. I remember one conversation we had that was both about preservation of data and decision making but also the impact of that kind of privacy preserving technology and changing relationships to privacy has on kind of future preservation. But on government records, I remember we were talking about the consequence of having outsourced lots of data related functions is that it can take preservation of that information out of scope for the recordkeeping authority. And that it’s kind of like, you know, when you are trying to do some research on a project and you can find a report that references some research and that’s what you have, but you have no access to the actual research underpinning it, you can’t look at the data for yourself. And that’s increasingly what I find with bumping up against is we can get the outputs, we can archive reports, we can archive snapshots.

EB: But the preservation of data that’s collected by private companies, but essentially acting as public services, it has been really difficult. It feels ? about privacy in a really interesting way, but that’s kind of a separate…

CF: Oh don’t worry. I will have a quick question about privacy before we finish.

In the next episode we talk do indeed talk about data and privacy, and Ellen offers some thoughts on where this is all heading.

About Ellen Broad

Ellen is Head of Technical Delivery, Consumer Data Standards for CSIRO’s Data61. She returned to Australia from the UK in late 2016, where she was Head of Policy for the Open Data Institute (ODI), an international non-profit founded by Sir Tim Berners-Lee and Sir Nigel Shadbolt. While in the UK Ellen became an expert adviser to senior UK government minister Elisabeth Truss on data. She has also held roles as Manager of Digital Policy and Projects for the International Federation of Library Associations and Institutions (Netherlands) and Executive Officer for the Australian Digital Alliance.

Ellen’s written and spoken about AI, open data and data sharing issues in places like the New Scientist and the Guardian, for ABC Radio National’s ‘Big Ideas’ and ‘Future Tense’ Programmes and at SXSW. Unrelated to her job, Ellen built an open data board game, Datopolis together with ODI CEO Jeni Tennison, which is being played in 19 countries. 

Recordkeeping Roundtable is made with Jetpack Podcast. Full show notes and every episode at https://rkroundtable.org/

Robot | by Sebastianlund

In this episode, we conclude our three part conversation with The National Archives UK’s John Sheridan, with a chat about machine learning and its possible applications in the management and use of records.

Our thanks again to John for being generous with his time and offering us so many interesting ideas. Remember to subscribe to the TNA blog for updates on the many interesting projects underway there.

You can subscribe to this podcast series on Google Play; iTunes is having a little moment with adding new podcasts but hopefully soon.

Transcript

CF: Now, we have a short period of time remaining, so I hope I can continue for another little while with the conversation, if that’s all right with you, if you’ve still got the time.

JS: Absolutely.

CF: Excellent, very good. Because this third subject is one that I think I’ve certainly noticed being discussed more and more within our profession, and indeed, there’s been some interesting experimentation happening in Australia by some former colleagues of mine. I’m talking about machine learning. So, for example, State Archives and Records New South Wales recently reported on an experiment that they ran, and I saw that the National Archives has blogged about your own experimentation. So I guess this is a pretty broad question, to find out what it is that you’ve been doing to test out machine learning and its applications for management, searching and use of records.

JS: So our primary focus is just to develop our own knowledge and understanding as an institution around machine learning approaches, capabilities, what kinds of things is this good for, what’s relatively easy for us to pick up and use. So we’ve done things like, we had inside the archive a two-day machine learning Hackathon, where running into the Hackathon, we gave primarily technical people working at the archive like a primer into the main machine learning algorithms and some of the tools you can use, some of the software libraries you can use if you want to develop a machine learning project or a machine learning-based project, and then just have people working in teams to create their own project.

And we had a really amazing variety of candidate applications, whether it was trying to solve the problem of which research guide do you put against which part of the catalogue and can you use machine learning to… And we’ve got like 500 research guides, and 30 million catalogue descriptions. We’ve never been able to work out which research guide you might want to offer a user when they’re looking at a given catalogue description, and the catalogue’s too big for us to be able to manually do that. So that could be trying machine learning as a way of doing that and that actually works pretty well.

CF: Okay, so potentially you could have a bot that popped up and said, “Oh, I see you’re interested in records of X, Y, Z. Here is a guide.” Would that be one potential use of that?

JS: Right, exactly right. With some kind of probability. So that turned out to be like a just a quick experiment, but that looks a viable area for us to explore. On the digital preservation side, we’re really interested in whether we could use machine learning to help us characterize digital objects that we can’t characterize using file format signature identification. Typically, this would be things like computer programs, they all report as text files, and signature identification is not a good way for being able to distinguish between whether you’ve got computer software that’s written in Java or C# or Python, say. And we’re particularly alive to the fact that in government, we see people increasingly producing things that have all of the characteristics of records that are computer programs or mixtures of things that are computer programs and textual content.

So, there’s a thing called R Markdown, which statisticians are using increasingly inside government, where they’re mixing Markdown textual content with statistical algorithms in R in single assets. These things have sure have many of the characteristics of records that we think are likely to be appraised and selected, so that’s a mixture of textual content and computer code. Now, signature identification is not good at identifying that kind of content. So could we use machine learning and a training set to start identifying some of these object types that signature identification doesn’t work for? And it turns out, yes, actually that works pretty well if you have a good training set. So we learnt that.

We’ve also tried machine learning, and this is where I think the strongest interest is for decision-making about selection. Now, we know that there’s a real challenge for identifying records from non-records. The classic challenge, of course, is around email, and we know that machine learning is pretty good at solving classification problems, so you can do things like, “Is this personal email? Is this business email?” And you can set up a machine learning-based approach for giving you a probabilistic insight into, “That’s almost certainly personal email, that’s almost certainly business email, that may be some kind of combination and I’m only 60% sure.”

We are, I think, reasonably confident that the future digital record of government is going to be selected through machine learning, and that that is the future for digital archives from here on in. There are some problems that current machine learning capability is well disposed to being able to address in that context of using machine learning for record selection, and business/personal email is a classic case in point. There are some things which are super hard. If the decision-making relies on knowledge of the wider context, then it’s really difficult to see, without having some kind of model for that context, how a machine learning system is going to be able to be helpful. So we’re really keen to try and understand some of those parameters and then get a sense for where do we need to lean on I suppose more traditional forms of knowledge engineering alongside some of these newer machine learning-based approaches. And that’s very much our focus, but initially, it’s very much about just building our own appreciation and know-how as an organization so that we can experiment and ask and answer some of these kinds of questions.

CF: Yes, the questions around the use of machine learning to make significant decisions affecting government services, or individuals that are happening in the wider community. I guess archives need to start to think about and take ownership of what implications are there for using these technologies in terms of things like trust and things like perceptions of bias. So transparency, I suppose, is partly the solution to that. Do you have any thoughts on those kinds of wider societal perceptions of an archive that uses machine learning to identify and manage records of its societal memory?

JS: So there’s two elements to this. In terms of the possibility, then machine learning provides the only really scalable way, I think, that we’re going to arrive at decision-making for selection of the record that needs to be permanently preserved from the information that can be discarded or left behind. And typically, government departments in the UK, and we see this being a pattern elsewhere, have a heap, a digital heap, from which the long-term public record needs to somehow be extracted, permanently preserved, and made available. And when you’re talking about tens of terabytes or maybe even hundreds of terabytes of heterogeneous digital records, e-mails, and Word documents and all sorts of other things, the only way that’s going to get done is through machine learning.

It may even be that a machine learning approach to selection is more consistent, more reliable, and can be more transparent than the historical approach to how we’ve gone about it, which has been human decision-making. But we are going to run into some hard limitations, where machine learning is good for classification problems, but it’s bad for having an appreciation of a textual understanding, and whether we can find ways of training systems that can give us some proxies for contextual understanding is stuff that we need to explore and research. On the flip side, we can also see that machine learning systems are going to be increasingly widely deployed by government agencies and that we can imagine deep networks themselves becoming public records, so we need to figure out how on earth might we appraise and just select a deep network and preserve it and make it available.

So we have that element too, and again, the only way that we even begin to get an appreciation for what that means is by rolling our sleeves up, developing our understanding of the technology, trying it out, learning and asking these questions and trying to answer them. It’s something that we’re clearly too small to do this just on our own, it’s something that I think archives around the world need to be developing all of our knowledge and our understanding and capability and having these conversations if we’re to really be able to build the body of practice that we clearly are going to need.

CF: Yes. Well, it’s such an interesting area, and I’m sure when you attend iPRES later in the year, you’ll probably run into some conversations on the experimentation that’s happening in other sites as well, and hopefully that can bring about some more partnerships and collaboration in this space, because it’s such an important one.

JS: I hope so.

CF: I think, perhaps, I need to draw this fascinating conversation to a close. And so I want to say thank you so much for giving your time today and so thoughtfully addressing all of those topics, it’s very much appreciated. So thank you so much, John Sheridan from the National Archives UK.

JS: Thank you very much.

About John Sheridan

John Sheridan is the Digital Director at The National Archives, with overall responsibility for the organisation’s digital services and digital archiving capability. His role is to provide strategic direction, developing the people and capability needed for The National Archives to become a disruptive digital archive.

John’s academic background is in mathematics and information technology, with a degree in Mathematics and Computer Science from the University of Southampton and a Master’s Degree in Information Technology from the University of Liverpool.

Prior to his current role, John was the Head of Legislation Services at The National Archives where he led the team responsible for creating legislation.gov.uk, as well overseeing the operation of the official Gazette. John recently led, as Principal Investigator, an Arts and Humanities Research Council funded project, ‘big data for law’, exploring the application of data analytics to the statute book, winning the Halsbury Legal Award for Innovation.

John has a strong interest in the web and data standards and is a former co-chair of the W3C e-Government Interest Group. He serves on the UK Government’s Data Leaders group and Open Standards Board which sets data standards for use across government. John was an early pioneer of open data and remains active in that community.

Recordkeeping Roundtable is made with Jetpack Podcast. Full show notes and every episode at https://rkroundtable.org/

In this episode of Recordkeeping Roundcasts, we pick up the conversation with TNA’s John Sheridan, turning to the experimentation that he and colleagues at allied institutions are doing in distributed ledger technologies.

For background, take a look at this recent post by Alex Green on the TNA blog about Project ARCHANGEL.

Transcript

CF: It is a critical juncture, I think, for archives in all forms. Certainly, as you’ve articulated, the expectations for digital records, digital information, discoverability are exerting pressures on archives. I need to move on, as I could talk about this stuff all day. I need, tragically, to move on to the next subject, which is linked to this notion about reputation, in that another project that you’ve been engaged in recently has to do with trust, and this is an area that I’ve been very interested in, in that archives, I feel, for too long have relied on the either legislated or culturally allocated notion that they are trusted institutions. And in some cases, I think it’s allowed a certain laziness to creep in around things like demonstrating the work that goes into maintaining authentic reliable records.

There’s a sort of a veil is drawn across some of the processes, the mysteries that go on behind the walls of the archive, and that’s been to the detriment, I think, of archival institutions. And so, it’s terrific to see the National Archives, for example, being so open about what it’s doing. But in relation to this project, for the listeners of the podcast, this is a project called ARCHANGEL, which is breaking new ground by using blockchain technologies to allow verification of born digital records over any timespan, really. The data is permanently preserved using this technology, which relies on peer to peer distribution and consensus checking, and thus removes the need for a trusted third party. I think it does offer some very exciting opportunities for archives to move from this institutional trust by mandate to technologically enabled transparency of processes on record. So I wonder, could you just briefly tell us a little about the project, and I’m interested to know if you see it as having potentially wider application in terms of record-keeping solutions outside of the traditional archival sphere.

JS: So it’s a fascinating juncture. The arrival of blockchain technology, I think, is important for anyone who’s interested in record-keeping. This is a new record-keeping technology, fundamentally, and I think it’s The Economist back in 2015, described blockchain as the trust machine. So a big part of our motivation, and partly because we have a concern about just leaning on an institutional basis of trust, which works fine when your physical collection is effectively immutable by virtue of its size and by virtue of its physical properties, but your digital collection is fundamentally changeable, and in the case of born digital records that are closed for a significant period, we’re essentially asking people to trust that the record is the thing that we’re claiming it to be over quite long periods of time.

Now, we are absolutely not complacent about the institutional basis of trust. We think we need to augment that with new archival practice, and blockchain is a really interesting candidate technology for how we might go and do that. It relates quite closely, though, to the previous conversation we were having about context, because whereas blockchain may provide us with the technological means for assuring over time that a record hasn’t been altered, and essentially by computing a hash, so running an algorithm over a record and turning the record into a number, and you put that number into a blockchain, the blockchain gives you a date-time stamp for when you wrote that number onto this distributed ledger, and then we imagine multiple archives all participating in curating and sustaining this archival distributed ledger over time, and essentially guarding and guarantoring or guaranteeing each other’s collections. By writing these hashes to the ledger with the date-time stamp, it means that at a later date you can come back when you have the same object and you can re-compute the hash and you can verify, yes, this is the same thing that the archive said that it had at this previous point in time.

That isn’t only thing that you need to know about a record. You need to know who created it and when did they create it and what were they doing. So you’ve got this really interesting question of assuring the record and also trying to assure the context of the record, the archival context. There’s some interesting questions about how much of the information that you have about a record do you put on to the ledger? And the point of ARCHANGEL is to research all of that, to understand what information do you put on to a distributed ledger, from an archival practice point of view. If you have a digital archive based on the Open Archival Information System reference model, where in your process do you write these hashes onto a ledger? Where does it fit in the context of a digital preservation system? How long might you expect the hash to be valid for if you do this?

So are you talking about five years or 10 years or 20 years? What happens at the point where your hashing algorithm becomes provably breakable mathematically? How might you then move your ledger to a new form of encryption, a stronger form of hash? So all of these kinds of questions, what happens in the case where you’ve taken a content object like a video? And we are preserving quite a lot of video at the National Archives in the UK. And in order to preserve it, you have re-encoded it into another format that’s easier to preserve. Well, if you just hash based on the sequence of bytes, then you’re going to get a different hash, so are there ways in which we can create content hashes, as opposed to bitstream or bytestream hashes, to verify content objects through forms of migration?

So these are all really important research questions that we’re exploring through ARCHANGEL and it’s giving us the opportunity to try this stuff for real, roll our sleeves up, set up a distributed ledger. If anyone’s out there and wants to try working with a distributed ledger, just get in touch, because we can really quickly get you up and running as a node potentially, on our ARCHANGEL research network of nodes in our archival distributed ledger, so you can experiment with some of this for yourself. Asking and answering some of these questions in order to get a sense of whether this truly is a technology that we can reliably use as part of our digital preservation infrastructure and that others can reliably use that is adding that extra, I suppose, cryptographic assurance ultimately that the records that we have are not being altered over time and you can prove it, you can cryptographically prove it.

CF: Is it being used and experimented in in agencies in the UK government? And I guess, you know, could you see some applicability of some of the approaches that you’re developing in the sort of earlier engagement and intervention piece that is mentioned in the strategy around identifying, registering records that are out in agency land?

JS: So, that’s a great question. And the UK government is for a lot of reasons very interested in distributed ledger technology. In London we’re obviously at the heart of FinTech. So, there’s a huge amount of work that happens around developing all kinds of novel or new or interesting financial technologies. So, the government has funded quite a lot of research in a wide variety of contexts. It’s been great for the National Archives, because we’re joining a UK government community of practice around distributed ledger technology, where we can get some visibility on how other people in government are looking at distributed ledgers for all of the use cases you’ll be familiar with, so things like land, property transactions or intellectual property rights assurance, through to things that from my perspective I’m not altogether sure I quite understand how they work, around health data or all sorts of use cases that are being explored.

It gives us the opportunity to swim in that sea. I have to say, the archival use case, from the ones that I see, looks like one of the strongest use cases for blockchain going. I feel that we’re really onto something for archives, because the jeopardy’s real.

So, it’s a great opportunity for us to, as I say, roll our sleeves up, explore what this looks like, see what other people are doing, learn from their experiences. There’s also, I think, an opportunity for archives, and this is… If we think about the value that an archivally assured distributed ledger might have, then we have an unrivaled prospect of longevity collectively as institutions. And so you may imagine, actually, there being some incentive for archives by allowing other kinds of transaction onto an archivally assured distributed ledger. And that may be a way in which the archives are incentivized to curate and continue to sustain a distributed ledger.

So, we’re very interested in whether there are some new economic models for archives through distributed ledger technology. Again, from a research perspective, but asking some of those questions, ’cause someone has to pay. Someone has to do the work that keeps the distributed ledger alive. We know that typically archives are pretty poor as institutions. So, is there some opportunity to trade our relatively high level of trust as actors and our unrivaled prospect of longevity into some form of incentive mechanism through distributed ledger technology that means that we get some benefit ourselves in terms of this strong assurance, cryptographic assurance for the records we’re preserving on one side, but we also have something that is economically sustainable, because maybe there’s a route for other people to be able to benefit from or use or work with the distributed ledger that we’re standing up. And again, in ARCHANGEL that’s very much part and parcel of what we’re trying to research.

CF: I wonder, there may be ways, then, that by having such services that are economically viable and desired by government organizations that there would be opportunities for the archive to get visibility, if not registration, of high-value classes of records that perhaps otherwise wouldn’t come into the the transfer process.

JS: Absolutely.

CF: So that might be another benefit.

JS: That’s exactly right. And there’s a very interesting use case that, again, we’re interested in researching around using smart contracts for timed release, so which again… I’m not suggesting that record creators are yet in a place where they’re yet willing to go, “Yes, let’s… ”

CF: Yes, yes, let’s automate…

JS: “Let’s automate digital transfer using… Automate it right up front, smart contracts, timed release 20 years, there you go.”

CF: Off you go, yes.

JS: We could potentially use that in terms of ourselves, where we have records that do have, that we keep closed with a point in time where they will be made available. And so you might imagine the archive itself using smart contracts for that. And we’re interested in exploring more generally timed release and whether that’s something that even… Is that something that we could conceive to make work. And again, it’s research, so we can do the technical assessment. We’re interested in doing the business assessment and finding the sweet spot where we actually understand how this technology can be viably and practically deployed in our context, against the wider context where the level of trust that we imagine there being in digital archives in future is significantly less, because of the general skepticism that is rising and rising and rising in any form of digital evidence. And it’s against that backdrop of a rising skepticism in all forms of digital evidence, for very good reasons, and I suppose a sense that the archive finds itself in an arms race with the forces of fakery that we think about privileging our collection with new forms of trust and then just pushing the light around understanding, well, what are the opportunities here.

CF: Look, it’s a fascinating project, and I’ll be keeping a close eye on it, and I know that you’ve mentioned that it will be the subject of an iPRES paper later this year, so I’m sure that many of us in the community will be watching with great interest to see what happens next.

JS: The other thing to say, we’re very much committed to working in the open, and we’re very keen to collaborate with other institutions. So if anybody wants to try being part of an archival distributed ledger as part of their research and dipping their own toes in the water, then we are well placed to be able to make that pretty easy to do. So just get in touch and we’d be really glad to have that conversation.

About John Sheridan

John Sheridan is the Digital Director at The National Archives, with overall responsibility for the organisation’s digital services and digital archiving capability. His role is to provide strategic direction, developing the people and capability needed for The National Archives to become a disruptive digital archive.

John’s academic background is in mathematics and information technology, with a degree in Mathematics and Computer Science from the University of Southampton and a Master’s Degree in Information Technology from the University of Liverpool.

Prior to his current role, John was the Head of Legislation Services at The National Archives where he led the team responsible for creating legislation.gov.uk, as well overseeing the operation of the official Gazette. John recently led, as Principal Investigator, an Arts and Humanities Research Council funded project, ‘big data for law’, exploring the application of data analytics to the statute book, winning the Halsbury Legal Award for Innovation.

John has a strong interest in the web and data standards and is a former co-chair of the W3C e-Government Interest Group. He serves on the UK Government’s Data Leaders group and Open Standards Board which sets data standards for use across government. John was an early pioneer of open data and remains active in that community.

Recordkeeping Roundtable is made with Jetpack Podcast. Full show notes and every episode at https://rkroundtable.org/

Welcome to Recordkeeping Roundcasts, a series of conversations with interesting people who are doing interesting things.

In this first episode the Recordkeeping Roundtable’s Cassie Findlay is talking with John Sheridan, Digital Director at The National Archives UK, about the challenges of scale and complexity that come with digital recordkeeping. As background for this conversation, take a look at TNA’s Digital Strategy, released in May 2017.

Transcript

CF: It’s terrific to be speaking to you, John Sheridan from The National Archives UK. Thank you very much for agreeing to chat with me about a number of topics of interest.

JS: It’s a real pleasure to have this opportunity to talk about all these great things that we have to think about.

CF: Yes I was really fascinated to read the National Archives’ digital strategy that came out last year, and a number of other posts to the blog that touch on many issues that I have been interested in myself, and that the people of the Recordkeeping Roundtable, my colleagues in Australia, have also been interested in, and so today what I wanted to do was dive down into some of those issues. I must admit it was quite hard to pick them, because we could have gone down any number of paths in this conversation, but perhaps I’ll start off with the first subject, which is in broad terms, it’s about scale and complexity. In the National Archives’ digital strategy, the idea of transfer and ingest of digital records is mentioned, and there is a commitment to standardize and streamline the process for transfer, talking about a self-service digital interface for government departments to automate the registration, transfer and ingest of new records. And there’s also some mention about changes to how digital records are described and the cleverer use of metadata.

So I guess my question to you, having worked myself in the very challenging and complicated world of accepting and ingesting and preserving digital records. These are fantastic goals, but are there limits to what can be achieved in this regard? And if so, where do they lie?

JS: That’s a great question. And of course, there are limits. I think, to give you a little bit of context for how we arrived at this place, we’ve been archiving digital records for perhaps as long as 15 years now, and we’re on therefore maybe a third incarnation of the digital records infrastructure and a digital archiving system. So we’ve quite a lot of experience at what it takes to manage the kinds of complex varied records that government is producing and that we’re needing to archive and make available, and our processes up to now have been quite, I suppose, hand-crafted. Each collection has looked very different, each EDRM system that we might take records from has thrown up different issues; databases look very different to managing emails from an EDRM system, which looks very different from managing a collection that’s been managed, say, on a shared drive.

And we’re trying to get from this point-to-point solution to a place where we essentially make digital transfer a commodity. It’s a thing that we can do readily and easily and efficiently. Now, we’ve been poking at what is it about digital transfer that causes us real difficulty, and the variation in source material and the variation in collection and the variation in issues that we discover when we try and ingest digital records is certainly one of the areas where we have… It generates quite a lot problems for us. It generates problems, particularly around things like consistency of metadata. So some of this is about creating a transfer system that’s much more forgiving for the information we have about the records, because we know that one of the most important risks that we can manage is the preservation risk, and the understanding that we have of a digital collection, and in order to be able to effectively manage that risk, we need to ingest it and we need to be actively preserving it.

Now, that brings in questions around automatic metadata creation, but also being much more forgiving about how accurate we need information to be and our expectations of the information that we have. So if you have something like a creation date, you can either be in a world where you absolutely require a creation date for a record to be perfectly accurate, or at least a plausibly accurate claim. That sets the bar quite high in a world where the records that you are confronted with are routinely reporting a date that you know must be wrong. So, how about you have an approach to how you think about the metadata that’s available, where you have an acceptance that there will be a variation in the reliability of that metadata, say, about a date of creation, and you base your decision-making against an acceptance of variation in quality, rather than setting the bar so high that the data must be right for you to be able to make a perfectly right decision.

And so it’s partly in order to have a transfer process that works in real life. It’s partly about streamlining and automating, but it’s also about, in our minds, having a more realistic expectation of the information that we’re likely to gather, particularly as we have more automated processes, and that we’re realistic then in terms of how we use that information for our decision-making. And it’s about, I suppose, a more general shift in terms of accepting and having archival practice for, an era where the record-keeping traditions at play by record producers in the digital era are just very different from the record-keeping traditions and practices that were at play in the era of physical records, and accepting that the digital record is going be much messier, it’s going to be more chaotic and we’re going to need to have different expectations in terms of the information that we have and that we use. If we are to have transfer processes that scale, rather than going back and fixing stuff up or putting an enormous burden back on record creators to can, just perfect metadata that they’re never going to be able to produce.

So it’s as much, I suppose, from my perspective, this idea of building a technological capability that enables us to automate or increasingly commoditize transfer, but also this acceptance about data quality and being much more realistic about our expectations of the data that we have. In a way, that’s actually quite… It’s quite uncomfortable for an archive because, frankly, your catalogue gets really badly messed up. Archival institutions are, yep, the born digital record of a 21st century government, is going to be more chaotic, the metadata’s going to be a lot poorer, and your catalogue is going to be much, much messier than you’re used to. And we just have to live with it.

CF: And I suppose, there’s fantastic opportunities, I think, in that chaos and complexity in the way that relationships can be drawn out between records and between agents or actors in the business, but it does stress and strain some of our existing descriptive practices, as well as our management tools. Do you have any sense that description at a higher level and letting the complexity play out and using tools to analyze and investigate records that are different to the traditional discovery tools is partly the solution, or are you looking to adapt descriptive practices down to the item within the item, within the item type of scenario?

JS: So how the archive goes about providing context, archival context, so that the user of a record can have some sense of its evidential value, based on some claims about the record creator and what the record creator was doing at the time the record was produced and something about how the record was managed, the record-keeping system that was at play, that’s all crucial, and it’s key to how the archive goes about providing value, how a digital archive goes about providing value. The approaches and the standards that we have for doing that, though, are, I would go so far as saying badly out of date, badly out of date. So the ISAD [G] type cataloging standard and its incarnation in technological form, with at the interchange format, the encoded archival description, this whole body of how archives go about providing context is serving us really poorly in practice, and it’s because we’re taking essentially approaches to description that worked well for physical records and we’re applying them to digital records and they don’t work so well anymore.

So we need to break free of that, but we still have to… We have to be able to give an answer to the user of the record, which is the key questions, who created this record? What were they doing? How was this managed? What was this part of as a system? It’s just the answer isn’t going to be a conceptualization against original order in a hierarchical descriptive inventory. It’s not going to look like that and it shouldn’t look like that. It’s going to be something that’s much more fluid, it’s going to be connected, related concepts in something that looks more like a graph-based data model than a hierarchical model, it’s going to be more data-oriented rather than textual description, and the ability that we have, as you say, to compute over the records means that we will be continually through actively computing over the collections that we have, the digital collections that we have, we’ll be continually identifying new connections between records.

The other piece where there’s just the most extraordinary opportunity for digital archives is that the digital record of government becomes contemporaneous with the World Wide Web, this massive globally deployed information infrastructure, which itself is being archived, and so for the digital archive in a way that it was never true, or the possibility was never there for us to realize of how the archive is contextualized with the rest of what’s known and available in the world. For a digital archive that is on the World Wide Web, and part of a globally deployed information infrastructure itself, which has been significantly archived. So I’m thinking, for example, of connecting email records in 1997 for a key decision that was made inside the UK government. Now, 1997 is the same year that the BBC launches its website, and you can see the news stories that the BBC was presenting in 1997 to this day. So you can compare what a politician was saying to the BBC or what they’re reported as saying on the Web in 1997 and use that as a piece of context for the exchange they may have had about the same issue with, and the briefing they may have received ahead of that interview, which would be part of the digital public record.

Now, this is a wholly new opportunity and, again, our approach to how we provide context, that we design it in a way that enables those connections, those linkages, enables something that’s much more fluid than the traditional approach that we’ve had through the rigid hierarchical descriptive inventory. That’s the prize. The problem is, is it’s a long way to go from where we are, and we don’t want to, if I use the cliche, throw the baby out with the bathwater. We’re looking to try and do something that is, that realizes some of these possibilities, but still means that we are retaining intellectual control over our collection and that we’re able to answer the most basic questions around archival context.

We’ve said quite a lot about it in our guidelines for cataloguing born digital records, but this is very much a work in progress, and I think it’s something that digital archives around the world are going to have to work on and be much more radical about. And we’re going to have to work hard on this over the next decade. We’ve a long, long way to go to having the kinds of practice around archival description and context that we need. I think from the National Archives’ perspective, we are putting ourselves out there saying, “The old world will not do. This is just not going to work anymore. It’s not going to scale. It’s not going to meet users’ needs, and it doesn’t allow us to realize the possibilities that we have at this juncture.” Archival context is still key, but we have to now move on and re-imagine it, and we’re trying to do this very much by working in the open. It’s why we’re blogging and talking about some of our ideas so openly.

Next episode: We talk with John Sheridan about blockchain experimentation and the ARCHANGEL project.

About John Sheridan

John Sheridan is the Digital Director at The National Archives, with overall responsibility for the organisation’s digital services and digital archiving capability. His role is to provide strategic direction, developing the people and capability needed for The National Archives to become a disruptive digital archive.

John’s academic background is in mathematics and information technology, with a degree in Mathematics and Computer Science from the University of Southampton and a Master’s Degree in Information Technology from the University of Liverpool.

Prior to his current role, John was the Head of Legislation Services at The National Archives where he led the team responsible for creating legislation.gov.uk, as well overseeing the operation of the official Gazette. John recently led, as Principal Investigator, an Arts and Humanities Research Council funded project, ‘big data for law’, exploring the application of data analytics to the statute book, winning the Halsbury Legal Award for Innovation.

John has a strong interest in the web and data standards and is a former co-chair of the W3C e-Government Interest Group. He serves on the UK Government’s Data Leaders group and Open Standards Board which sets data standards for use across government. John was an early pioneer of open data and remains active in that community.

Recordkeeping Roundtable is made with Jetpack Podcast. Full show notes and every episode at https://rkroundtable.org/