In this episode of the Državljan D podcast, we dive into the challenges and hidden dimensions of modern security policies with one of Europe’s leading investigative journalists, Giacomo Zandonini.

For years, Zandonini has uncovered how extensive surveillance systems, data infrastructures, and technological ecosystems are being built in the shadows of EU institutions—shaping the future of security and its impact on fundamental rights.

His latest investigations highlight Europol’s rapidly expanding powers, its growing ties with the security-tech industry, and rising concerns about transparency and oversight.

00:00:10 Domen Savič / Citizen D 

Welcome, everybody. It’s the 14th of November, 2025, and you’re listening to this episode of the Citizen D Podcast, airing on the 15th of November, same year. Today, we’re joined by Giacomo Zandonini, an investigative journalist whose work has exposed the unusually close and often opaque relationships between Europol and major tech companies. Welcome, Giacomo, and thank you for dropping by.  

To start us off, can you recap your investigation? There’s a link in the notes, but for people who don’t read, can you say a few words about what was the focus of your investigation into Europoling connections with tech companies and what are some of the main findings? 

00:00:59 Giacomo Zandonini 

Yes, thank you very much, and thanks for having me here, Domen. 

The investigations I worked on with the UK-based group Statewatch stemmed from a recent initiative by Europol called the Research and Industry Days. It’s an annual meeting held at Europol’s headquarters, where industry representatives—mainly from various companies and a few research institutes—are invited to present their latest technological tools to Europol. 

This event is organized through an invitation process that takes place around this time each year. For example, the call for the 2026 edition has just been opened. Europol publishes a sort of wish list outlining the types of tools and sectors they’re interested in. 

Since the Research and Industry Days were launched in 2024, the focus has largely been on advanced technologies—AI-powered and machine learning-based systems. These range from tools for monitoring social media and the broader internet, to network mapping and analysis, as well as the use of drones, automated devices, and systems for processing, analyzing, and extracting data from phones or other devices. 

Many of these tools can be considered highly intrusive in terms of privacy, depending on how they’re used. They certainly have that potential. I chose to focus on this annual event because it offers a clear window into the relationship between Europol and the tech industry. It seemed like a good starting point to understand how these interactions work. 

From there, I expanded the scope of the investigation to provide background and context on how this relationship is regulated and how it has evolved—particularly over the past decade, during which Europol’s ties with industry have been notably close. 

That’s the main contribution I aimed to make. 

00:05:02 Domen Savič / Citizen D 

To riff off your comment about how opaque these relationships often are—at the same time, Europol is essentially putting together a wish list that reflects the major selling points the tech industry is currently pushing out to the world. 

You mentioned AI and other tools… So, who would you say is actually steering this relationship? Who’s in charge? Is it the corporations, offering these tools to public decision-makers or are public decision-makers genuinely choosing which tools to adopt based on real needs and requirements coming from the field? 

00:06:00 Giacomo Zandonini 

That’s a very good question, and I don’t think I have an exhaustive answer—because it’s really a two-sided story. 

Europol is an expanding agency within the European Union. Its annual budget isn’t as large as, for example, Frontex. For the past few years, it’s been around €240 million annually. The largest portion of that budget goes toward salaries for roughly 1,000 staff members, who are employed under various types of contracts. Beyond personnel, a significant share is spent on equipment and infrastructure—particularly IT systems. This includes the tools, software, and hardware Europol uses to transform itself into the EU’s Criminal Hub and Criminal Intelligence Analysis Center. 

So, the push to adopt industry products and maintain relationships with tech companies is central to Europol’s initiatives and programs—especially since 2015, when terrorist attacks struck the EU and France. We’re approaching the 10-year anniversary of the Bataclan attack, which marked a turning point for policing in Europe. That event triggered a major push for agencies like Europol to accumulate data and find ways to analyze it—primarily through software programs that, in recent years, are increasingly based on AI and machine learning. 

In this context, Europol began developing relationships with the tech industry. At the same time, companies became more proactive in pitching their products to Europol. One notable example is Palantir, a company revealed through investigations by two colleagues to have worked closely with Europol. Palantir’s software—particularly Gotham—is widely known and has been used across various security domains, from the war in Ukraine to Palestine, and within the U.S. security apparatus. It’s also present throughout Europe. 

Europol used Gotham to analyze and process its accumulated data. However, the relationship didn’t go smoothly, and eventually Europol discontinued its use of Palantir. That said, it’s still unclear exactly where things stand today. 

After 2022, Europol received a new mandate through regulatory reform. This mandate includes a focus on research and innovation, allowing Europol to develop systems internally—especially those based on artificial intelligence. They’ve since opened an Innovation Lab, which organizes events like the Research and Industry Days. 

Additionally, Europol houses the European Cybercrime Centre, which has been central to developing industry relationships over the past decade. So, while Europol is building its own in-house expertise, the industry remains essential. Europol actively seeks out AI-driven products, tests them, and sometimes licenses or contracts companies to use these tools. 

00:11:24 Domen Savič / Citizen D 

From a privacy activist’s perspective, there’s often a tension between privacy and security. Whenever tools like biometric surveillance, IMSI catchers, or AI-based surveillance are proposed, there’s always a counterargument: “But what about our privacy?” 

That framing assumes that privacy and security are opposites—that these tools actually deliver on their promises, and that we must choose one over the other. But how do you view the idea that this sense of security might be misleading? Do you think there’s a lack of evidence showing these tools are truly effective in delivering security? Could it be more about the appearance of safety—what some might call “security theater”—rather than actual improvements in public safety? 

00:12:50 Giacomo Zandonini 

Absolutely, that’s a key part of Europol’s relationship with the tech industry—especially when it comes to reusing tools developed by private companies. 

It’s important to remember that these companies are driven by profit, not by a mission to provide public security—at least not in our capitalist societies. Profit motives also mean aggressive marketing, and in this case, the primary clients aren’t citizens, but law enforcement agencies. These agencies often have access to significant funding, including from the European Union and the European Commission, to acquire and test new technologies and participate in research and innovation projects. 

There’s a broader movement pushing for law enforcement to adopt the latest tech tools to analyze data and identify potential criminal activity. But the risks are substantial. These systems often have vulnerabilities that can compromise the security of law enforcement agencies themselves. This has been well documented by scientists, researchers, and technologists. They also come with biases—facial recognition is a clear example—and high error rates. More concerning is that they can be used in ways that bypass democratic oversight. 

This is especially relevant when looking at the companies that participated in the last two editions of Europol’s Research and Industry Days. There are major players and smaller firms—some offering innovative tools, others partnering with larger corporations. 

One of the things I focused on was examining the track records of these companies and how their products have been used in other contexts. Of course, that doesn’t mean Europol will use them the same way, especially given existing EU regulations. But it’s still telling. 

Take Cellebrite, for example—an Israeli company publicly traded in the U.S. It’s one of the most powerful providers of technology for extracting data from phones, even without passwords. That’s their most well-known product, but they also offer platforms for analyzing and aggregating data, all aimed at law enforcement. 

Cellebrite’s tools have been used in authoritarian regimes and conflict zones to suppress basic freedoms. It’s one of the most controversial companies in this space. Even in Europe, there have been troubling cases. In Serbia, their tools were used to target journalists and activists. In Italy, they were used to extract data from activists’ phones—though that case is less well known. 

These companies are now central to modern policing. The problem is that we have virtually no public information about Europol’s relationships with these companies. In contrast, U.S. government procurement data—covering even secretive agencies like the FBI and ICE—provides at least some transparency. You can find contract amounts, product types, and renewal dates, even if the full content of the contracts isn’t disclosed. 

Europol, however, operates in a regulatory gray area. Its procurement and financial rules allow it to withhold contract details from the public under the justification of national or European security. Experts in transparency and EU law have told me this is highly problematic. These contracts should be subject to scrutiny. 

While it’s understandable that a law enforcement agency like Europol needs some level of confidentiality, its relationships with private companies raise serious concerns. These companies may access sensitive data and often have interests that don’t align with those of the European Union—or the security and rights of its citizens. 

00:20:49 Domen Savič / Citizen D 

To follow up with a question—you’ve been focusing on Europol and its lack of transparency for quite some time. I’d like to hear your thoughts on why transparency seems to be such a burden for organizations like this. 

As a reporter, whenever you investigate intelligence or police agencies, you often run into issues with their lack of transparency—whether it’s about procedures, tools, or the effectiveness of those tools. Their usual justification is that revealing too much could help the “bad guys,” so to speak. But do you think that’s the whole story? Or do you see deeper issues behind this culture of secrecy? 

00:21:55 Giacomo Zandonini 

Yes, I think this issue is deeply rooted in police culture. Of course, within law enforcement, there are many different personalities and perspectives. Some professionals and officials are more inclined to support transparency, and there are national norms across Europe that vary. 

But broadly speaking, there’s a prevailing idea that police institutions can be exempt from transparency. Formally, Europol isn’t a law enforcement agency, even though we often describe it that way. It’s an EU agency tasked with supporting national law enforcement bodies. Over the past decades, it has worked to legitimize its role by developing strong data analysis capabilities, and now it’s moving toward using advanced machine learning systems. 

National police forces send data to Europol, which analyzes it and provides feedback—essentially offering data insights as a service. Of course, it’s more complex than that, with joint projects, operations, and investigations also in play. Still, Europol seems to have absorbed the broader mindset that police work should be exempt from standard transparency obligations. 

This becomes especially problematic when private companies are involved. Europol often refuses to release information or grant access to documents held by EU institutions. The reasons given usually fall into two categories: commercial interests and national security. The commercial argument is understandable—a company shares internal information and doesn’t want competitors to know how its systems work. That’s fair. 

But the national security argument is used much more broadly. Europol often claims that revealing details would help criminals, who are supposedly watching closely. Sometimes, this logic implies—perhaps unfairly—that journalists investigating these issues are somehow aiding criminal organizations. 

The idea that transparency about relationships with private actors would benefit criminals is problematic, especially when applied so broadly. My team of journalists recently published a story on Europol’s AI program. We submitted a large number of access-to-documents requests, and most were denied. We now have five formal complaints pending with the European Ombudsman, all related to this lack of transparency. 

This matters because past investigations and questions from well-informed members of the European Parliament have revealed conflicts of interest—cases of “revolving doors,” where individuals move from Europol to private companies while continuing to work on the same issues. These are public officials with significant responsibilities, and such transitions raise serious concerns. 

So yes, some level of transparency is essential. I’m not saying Europol should disclose every operational detail or how a specific product is used in every situation. But we should at least know which companies they engage with, under what terms, and through what procurement processes. 

In fact, national police forces across Europe often provide more information about their contracts with companies than Europol does. That’s not a statistical claim—I haven’t reviewed every procurement system—but in countries I’ve looked at, like Italy, I’ve found clear traces of contracts with companies like Cellebrite, including renewal dates, contract amounts, and the types of software licensed. 

So yes, there’s definitely room for improvement 

00:28:34 Domen Savič / Citizen D 

We’re slowly wrapping up, and I just have two more questions for you. 

The first one relates to an investigation—or rather, a collaborative investigation—you were involved in back in 2022. I’m referring to a Guardian article titled “A Data Black Hole: Europol Ordered to Delete Vast Store of Personal Data.” It describes how Europol was holding a massive trove of data, much of it gathered from hacked encrypted phone services and even from asylum seekers who had never been involved in any crime. 

Looking at that story and comparing it to your more recent work on Europol, would you say the trend is moving in the right direction when it comes to privacy awareness within Europol and similar organizations? 

Or is this just history repeating itself—more privacy-invasive tactics and the same kind of opaque, non-transparent operations? 

00:29:52 Giacomo Zandonini 

I’d definitely say it’s the second option—it’s history repeating itself. 

Some of the issues raised by the European Data Protection Supervisor in early 2022, which we also covered in that Guardian article, were addressed to some extent. Europol had accumulated large amounts of data not necessarily linked to criminal activity—which means it could have included data on any of us. 

They did take steps to resolve part of the issue, at least from what I and other colleagues understand. For example, they began categorizing data more carefully. Europol is not allowed to retain personal data unless the individual is a suspect, a witness, or a potential victim of a crime. Any other data must be deleted. 

There’s also the matter of data retention—how long Europol can store personal data and the need to regularly review that. The problem is that Europol’s power to process, use, and store data is expanding. But we’re not seeing a corresponding expansion in oversight capacity. Despite what’s outlined in EU regulations, the European Data Protection Supervisor likely doesn’t have the resources or authority to fully monitor Europol’s activities. The internal offices within Europol tasked with oversight also raise questions about whether they have sufficient capacity, independence, and resources to do the job properly. 

So yes, we’re still facing the same challenges. And this lack of transparency could become even more problematic. Next year, the European Commission is expected to propose a reform aimed at turning Europol into what they call an “effective law enforcement agency.” While we don’t yet know the full details of the reform, one thing is clear: the proposed budget is expected to double—which is a significant signal of the direction things are heading 

00:32:46 Domen Savič / Citizen D 

To wrap things up—and maybe take the conversation into a darker direction—how do you see this whole issue? Europol is collecting massive amounts of data and using AI tools to analyze it and make decisions based on that analysis. 

How do you view this entire field in relation to the concept of digital sovereignty? It’s a term that’s been thrown around quite a bit in recent years, and I’m curious how you interpret it—especially considering that most of the companies involved aren’t European. We’re talking about U.S. companies, Israeli companies—essentially foreign actors handling EU data. 

00:33:47 Giacomo Zandonini 

Yes, that’s definitely an issue—and it’s becoming more visible. The European Union has launched strategies aimed at building digital sovereignty. One example is a recent proposal from the European Commission to create a dedicated European cloud for law enforcement. 

But so far, what we’re seeing from Europol suggests that powerful companies from outside the EU still play a significant role in its infrastructure. It may seem symbolic, but the fact that Microsoft employees now have a sort of permanent presence within Europol’s headquarters is quite telling. It raises serious questions about tech sovereignty. 

This collaboration with Microsoft could be a topic for further investigation, if colleagues decide to explore it more deeply. As I mentioned earlier, companies like Cellebrite—an Israeli firm listed on the U.S. stock exchange—and Palantir, which has grown even more powerful in recent years, are key players in this space. So yes, there’s a real issue here. 

To be fair, Europol does appear to be trying to move toward greater digital sovereignty. Some member states, like Germany and possibly France, are pushing in that direction. These are countries with strong ties to national and private security industries, and they’re advocating for more European control over tech infrastructure. 

For example, when it comes to Europol’s IT systems, they are increasingly contracting European companies to build and maintain them. Still, the EU remains heavily dependent on U.S. and Israeli tech markets. These companies are dominant—not just in terms of software, but also infrastructure. It’s definitely something we need to keep a close eye on. 

00:36:54 Domen Savič / Citizen D 

Thank you very much, Giacomo, for joining us and sharing your insights on this important topic and best of luck with your future endeavors. 

And to you, dear listener—thank you for tuning in, we release one episode each month, so stay connected and catch the next episode of the Citizen D Podcast in December. 

Thank you again, Giacomo, and all the best. 

Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!

V današnji epizodi podcasta Državljan D se z Niko Mahnič pogovarjamo o avtomatiziranem odločanju – fenomenu, ki vedno bolj oblikuje naša življenja, pogosto neopazno in v ozadju.

Nika Mahnič raziskuje digitalne tehnologije in etike umetne inteligence, posebno pozornost pa namenja vplivu algoritmov na družbo, zasebnost in pravice posameznika.

Skozi svoje delo opozarja na pasti avtomatiziranih sistemov odločanja in raziskuje, kako lahko posamezniki in institucije ohranijo nadzor ter transparentnost v svetu, kjer odločitve pogosto sprejemajo stroji.

Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!

Aljosa Ajanovic Andelic is a Policy Advisor at European Digital Rights (EDRi), where he covers issues related to illegal state surveillance like spyware, journalists’ digital rights, and the effects of techno-solutionist and securitisation policies on minoritised communities and people on the move.

We sat down with him to discuss the current situation in the field of spyware, the role of nation states and private entities and the role of European union.

00:00:10 Domen Savič / Citizen D 

Welcome everybody. It’s the 15th… No, it’s the 4th of September 2025 and you’re listening to this episode of Citizen D podcast on the 15th of September same year. W ‘re back after a prolonged, let’s call it a vacation and with us today is Aljoša Ajanović Andelić, policy advisor at European digital rights EDRi, where he covers issues related to illegal state surveillance like spyware, journalist digital rights and the effects of techno solutionism and securitization policies on minoritized communities and people on the move. Welcome, Aljoša, it’s good to have you here. 

00:00:55 Aljoša Ajanović Andelić  / EDRi 

Thank you. Thank you for having me.  

00:00:56 Domen Savič / Citizen D 

Let’s start with the opening salvo. We’re going to be talking about the spyware issue and I was wondering, why is spyware getting so much attention in 2025? Like, if you followed this this topic like a regular person, not like a digital activist, you’d hear probably something about Pegasus a few years back and then everything went sort of silent. But now, 2025, spyware is literally everywhere. Why is that? 

00:01:32 Aljoša Ajanović Andelić  / EDRi 

Well, the answer to that question, I would say that that is because of scandals. So that would be the same answer if you had asked me five years ago because nothing has changed. So from time to time, we come across these big scandals of illegal surveillance to different types of activities to political opposition in different parts of Europe, and that is when this topic always comes back to the front pages.  

In 2025, the biggest scandal has been the Paragon scandal, which was a revelation earlier this year that said that more than 100 people were spied across Europe with this spyware called Graphite that’s developed by the Israeli company Paragon. And this is something that that has made the front pages, because in Italy, some of those victims whose identity hasn’t been fully disclosed, but three or four of them have come, have come to the front and it happens to be journalists and humanitarian aid activists in Italy who have been spied illegally by… we don’t know who. So what happened?  

After these revelations came and these people spoke as one of the affected victims is what we have seen in all other scandals. So we’ve seen that in the past in Catalonia, but also in Poland and Hungary, which is that first the state denies all type or any type of responsibility on that and then starts considering a bit. Well, yeah, we spied on these ones, but not on the other ones and so forth.  

This is where we stand at this point in Italy in in which, you know, they have recognized that those humanitarian workers from the NGO Mediterranean have been actually spied by the state because they say that their activities qualify as crimes because they have been allegedly promoting human smuggling, but in the case of the journalists, the Italian government is saying that they are not involved, even though all the evidence points to that direction.  

What is happening in 2025, again, is that many scandals are unfolding. There’s also another case in Serbia that was revealed in which the Serbian Government has developed its own spyware. So it’s not just private companies, it’s also state developed spyware, and it was used in this case to also go or investigate the phones of journalists and also politicians. 

What makes this issue so problematic, or why should everyone, even if you are not a digital rights activist, why would you care about this? It because the fact what these programs do, which is to be able to access anything in your phone. 

This is something that’s very appealing to any common citizen, because we all now rely massively on our phones, they are kind of where all our information is stored and we all can imagine what kind of privacy violation would that be: that the state accesses you. 

00:04:48 Domen Savič / Citizen D 

You’ve mentioned the state several times. Are they still the main users or the main actors in this field or are there other actors? Maybe somebody from the private sector, from the criminal underground that is also a part of this spyware ecosystem? 

00:05:11 Aljoša Ajanović Andelić  / EDRi 

Well, mainly with the biggest issues on particularly spyware have come all of them from state authorities, private companies that are the biggest sellers of this.  

So, we can think about NSO group, but also Paragon and others, they all claim to just be selling to state authorities, so it doesn’t need to be the state itself, sometimes is police force. Sometimes is the judiciary power, but it’s mainly addressed to state authorities and the main scandals that have come, they all come from targeting from States and we do believe that’s the most worrying practice and the one that we as human rights defenders to care the most, because the state is the one that should be protecting your privacy and your digital rights and not attacking them.  

We’ve also had some cases in in countries like China or Russia in which it is not the state itself that conducts these attacks, but mercenary hackers or groups of hackers who are committed to the state in some way because the state is sometimes paying for their services or is servicing them with the programs, but they are not directly part of the state. 

 So we’ve seen that groups of mercenaries using these type to conduct smear campaigns or to intimidate certain groups.  

One good example is the regular minority in China, which has been attacked massively by these types of mercenary groups, but yeah, I would say that the biggest problem we’re facing is when states use that because when private, you know, criminals could be using that, that’s something that of course we should worry about, but what we cannot tolerate is that the state is using this without our knowledge and without any transparency. 

And of course you know, bringing to this union of companies with very worrying practices such as the set and the subgroup with state action, which is, for example, the police or law enforcement investigation, which is something of a public service so to say, and then it’s tied up with these companies that have virtually no oversight, so we don’t know what’s happening. 

00:07:36 Domen Savič / Citizen D 

You’ve mentioned transparency, right, and seeing how nation states are still the biggest or the most important actor in the field. How do you see this conflict of interest on the issue or on the on the level of transparency? So on one side you want your state or your elected leaders to be transparent about the activities, at the same time I’m guessing revealing too much information about the usage of spyware by nation states could harm their defense strategies and police tactics. 

Is there any middle ground on this issue so that at the same time the state still has this competing advantage against legal usage of spyware in order to prevent crime or terrorism and at the same time to calm the people down so they’re not using it indiscriminately, for, you know, tracking journalism journalists, activists and other parties? 

00:08:51 Aljoša Ajanović Andelić  / EDRi 

Well, that’s a very good question. So first of all, I would like to start by saying that when it comes to this type of spyware—so we’re thinking Pegasus, Graphite, and all these intrusive spyware that can gain control of your device, see all its historical logs, but also see where you are, activate your phone, activate your camera and so on—for this type of capabilities, we as a digital rights organization struggle to see how this can be compliant with human rights at all.  

To break down one human right, such as the right to privacy, you need to make sure that what you’re doing is necessary and proportionate. And this type of capability is completely unable to be proportionate because once they access your phone, they can access everything. So not just the conversations, let’s say, that can be necessary to get to know the activity of someone contacting criminal activity—so to say, it’s not accessing just that. It’s accessing the whole of the phone without any limitations. And what’s more worrying, without any log for that. So we don’t know. It leaves no traces on the phone most of the time. It tracks the proof that it has been inside that phone, and also we don’t have any logs of which information has been extracted, where it went, and so on and so forth.  

So starting from this premise, with the current state of play, we would say that there is no lawful application of these types of tools. I would also say another thing, which is: we understand the national security angle. Of course, you cannot reveal everything if you want to make sure that you are keeping your population safe. But at the same time, what these scandals have shown is that national security has not been an excuse for all of them. We’ve had scandals, as I said, in Poland and Spain, which have been targeting, for example, political opposition or journalists. And the problem of using this type of programs is that there isn’t a log of what has been done.  

That means that maybe, for example, five years after the Catalangate scandal in Catalonia unfolded, we have no information about what data was extracted from those phones. Where is that data stored? Is it stored in the Spanish police servers? Does NSO Group, an Israeli company, have a copy of all that data from Spanish citizens? And what can they do with the data? With this lack of transparency at all levels—there’s no transparency on whether a state has acquired this, so procurement opacity, total procurement opacity—but there is also a total lack of transparency on its use.  

Therefore, we would say that to start, for the victims that have already come forward, they need to have this transparency in order to have their rights upheld and to have remedies for the violations they have suffered. But again, there is a lot more to ask from states. So first of all, we need to know which are all the contracts that countries have signed with private vendors, but also which tools they have developed in-house, let’s say. And also we need to make sure that the victims who have come forward have this right of knowing what has happened with their data. Because in these cases of journalists or political opponents, I would say that no one can disagree if we say that these are unlawful uses of spyware. 

 And even though someone might not hold this absolute stand as we do—which is that spyware should be totally banned—even those people could agree with me that these cases were not used for national security. So national security cannot be an excuse to not reveal these types of practices: what data was taken, from where, where it is, and which company was used to do that. 

And then there’s the security angle — like, the state keeping an edge in spyware use by not revealing too much. Because, you know, if you tell people how it works, or which version or type you’re using, you might end up warning the criminals too. 

So, is that something that resonates? Like, is that actually in place? Or do you think — as you already said — since we don’t have, what’s the word… any real stats on how successfully spyware is being used, we can’t really know if this is even working? 

Yeah, that’s definitely one of the key things — if we don’t have information about what’s happened in the last 10 years in Europe, we can’t really draw any solid conclusions. But we do agree — and you can check out the paper we wrote in 2022 on state hacking — in that paper, we outlined 11 steps that states should follow to lawfully try and hack into the phone of someone suspected of committing a very serious crime. 

But none of that is actually followed when it comes to spyware. And that’s partly because of the capabilities of the tech, but also because of the entire structure that enables spyware use — starting with this commercial market. It’s a super opaque market that thrives on opacity. These companies need secrecy to operate. 

What they do is look for vulnerabilities — in programs or operating systems we all use. Think iOS from Apple, or WhatsApp, or Meta apps. They actively try to find flaws in those systems so they can use them to infiltrate phones. And all of this is driven by demand from states, who turn to private vendors to access citizens’ phones. 

That creates a whole other market — the market of vulnerabilities — and this market moves millions behind the scenes every year. From the last 25 major vulnerabilities Apple found in its own system, 21 were used by spyware companies to infect phones. That shows how this whole thing is a major driver of the vulnerability market — and that puts everyone at risk. Because once these vulnerabilities end up in a company’s hands, we’re all potentially exposed. 

So, by allowing this private market to thrive, states are not only putting citizens in danger — they’re also putting themselves in danger. These same vulnerabilities can be used against them. They can target state officials, or even be used to extract secrets from European companies — which is a huge issue in this context where everyone’s talking about economic sovereignty. This can backfire in so many ways. 

Now, of course, states have always had secret services. They’ve always had tools to investigate and stay ahead of criminals — that’s part of what police are supposed to do, and that’s fine. But when it comes to digital rights and privacy, it shouldn’t be a carte blanche to access any phone, any time. That’s neither necessary nor proportionate. 

What we’re saying is: there are ways to hack phones that don’t violate human rights — if certain requirements are met. And that’s the direction things should go. Instead, what’s happening now is the opposite — the market is shaping how states use these tools, instead of states saying, “Let’s find a human rights-compliant way to do this,” and then turning to the private sector for help building it. 

But that’s not what happened. The industry created these monsters — tools that let you access anything — and states are just buying them. And we think that’s deeply problematic. Because even in democratic — or so-called democratic — countries, we’ve seen these tools used to avoid scrutiny, or to go after political opponents. Not just to fight crime. 

So they’re dangerous tools by nature. Which, for us, makes them incompatible with fundamental rights. And what needs to change isn’t just how states use them — the entire market and the whole concept of spyware needs to be rethought. 

00:19:03 Domen Savič / Citizen D 

And you’ve mentioned private companies, right? So, mobile phone or device developers — that’s basically the origin of this whole horror story, right? These zero-days, these issues and bugs in software and hardware that enable the development of spyware tools or spyware software. 

So how do you see the role of these companies in all of this? You’ve got Apple, Google, Samsung, and other mobile developers who, on one side, are very eager to patch and fix flaws when they’re notified. 

But do you think they’re doing enough — proactively — to sort of stave off the dangers of, yeah, the whole spyware economy, so to say? 

 00:20:04 Aljoša Ajanović Andelić  / EDRi 

Well, in this case, I’d say — as you might know — in our work as digital rights activists, we’re most of the time actively fighting against big tech. We have totally different interests, and we believe this whole structure of monopolization and oligopoly in the sector is really dangerous for all citizens. 

But when it comes to spyware, it’s true — we’ve seen some good practices from these companies. One example is the recent Paragon scandal. It came to light because WhatsApp — so, Meta — actually informed people that they’d been targeted. They found the vulnerability and let those users know. Apple, too, is now publishing a report every three months with all the bugs they’ve found and patched in their system. 

So yeah, it’s in their own interest to keep their apps as secure as possible. And that just makes sense — if people lose trust in their services or in the strength of their end-to-end encryption, they’ll switch to something else. That’s pure capitalist logic. But in this case, it actually helps us, it helps having these companies actively involved in finding and patching vulnerabilities, and being vocal about it. 

We’ve also seen, for instance, when the UK tried to push for mandated backdoors in some systems, Apple pushed back — hard. And in the end, the UK government backed off. Because not just experts, but even the companies themselves, were saying: forced backdoors are basically just vulnerabilities — they make devices less secure, and open the door to anyone who can find a way in. 

So in this case, yeah, I’d say there are some good practices. Of course, more could be done. But it’s in their interest to protect the integrity of their encryption — and that’s why, in some cases, the only information we have comes from these companies. Because detecting spyware infections is really difficult. If you’re a user who’s been infected, you’re not going to know just by looking at your phone. 

There are very few ways to detect it. There’s no proactive investigation by Member States, or by the judiciary. So we end up relying on organizations like Citizen Lab — who, in good faith, and because they care about this issue, are actively looking for infected users. They’ve even developed tools to help detect infections, but we also rely on the companies themselves to tell us who’s been targeted and whose devices have been compromised. That’s just the reality. 

And there is some legislation, like the Cyber Resilience Act — adopted by the EU a couple of years ago — that requires products on the market to actively search for and patch security flaws. So there’s at least some kind of framework pushing companies to be proactive. 

But still, we’d urge them to double down on those efforts. Because right now, when states aren’t willing to be an active ally in this fight, the companies are pretty much all we have. So that’s something we definitely recognize — and we do welcome all efforts from these companies to help detect who’s been compromised. 

00:24:03 Domen Savič / Citizen D 

Because that’s the issue in the end, right? It’s so niche, so specialized, so problematic — and at the same time, it’s commercialized. It’s literally available to almost anybody with enough funds. 

So it really begs the question: how do we move forward from here? Is everything that’s happening now — the legislative proposals, the patches — is this a genuine attempt, or maybe a valiant attempt, to actually prevent this from happening? 

Or is it all just a bit of, you know… whatever color you think spyware is — greenwashing, basically? Like, everyone knows this will keep happening, but all you really need to do is signal a little that you’re “against it,” while at the same time not doing nearly enough to actually, you know… stop it. 

00:25:10 Aljoša Ajanović Andelić  / EDRi 

That’s a very good question. And it’s funny because, in the case of spyware, there isn’t even an attempt to greenwash it. Like, no one’s even trying to say, “Yeah, we’ll address this.” What we’ve seen at the EU level over the last five years is just total inactivity. 

The only thing that’s happened is the Parliament set up an inquiry committee — and to be fair, there were some MEPs who were really involved, really pushing to denounce this and calling on the Commission to take immediate action. Because this is a major human rights scandal. 

That committee concluded its work two years ago. Its final recommendations were adopted by the European Parliament. And two years later? Nothing’s happened. 

They asked for things like setting up a European version of Citizen Lab, and proactive legislation to control the use of these tools. The Commission? Completely inactive. They promised they’d issue a communication — just a non-binding legal text — to analyze how spyware interacts with GDPR and data protection laws. And they haven’t even done that. 

At the national level, with this whole “securitization” agenda spreading across Europe, the argument we keep hearing is: “We need this to fight crime, so it’s fine.” And what’s dangerous about that is, first, it ignores the fact that this shadowy market puts all of us at risk. But second, and maybe more importantly, it shows a huge disregard for human rights. 

Even the known victims — people who’ve already been targeted and have been facing years of judicial inaction — even they haven’t had their rights restored or protected. And forget about future victims — there’s been no move to prevent it from happening again. 

We haven’t seen a single state that’s been caught up in one of these scandals come out with a strong effort to investigate, to expose what really happened, or to hold anyone accountable. Not one. And we haven’t seen any meaningful commitment from the Commission or from Member States to actually make spyware use compatible with human rights. That just hasn’t happened. 

And it’s worrying. Because right now, Europe is basically becoming a hub for these companies. There’s no legislation stopping them from setting up shop, selling these tools internally, or exporting them. 

The legal tools we do have — like the dual-use regulation — they’re failing. They don’t cover these kinds of advanced spyware tools. So what we have is a total lack of legislation. And weirdly, the Commission doesn’t seem bothered by it. 

Which is strange to me. Because I compare this kind of spyware to someone breaking into your house. These days, all your most personal information — your whole private life — is on your phone, not in your living room. And yet, the Commission might come out and say, “Yes, we’re very concerned,” or “We’re taking this seriously,” but then they do nothing. 

I keep thinking: what if there were a pandemic of people getting their homes raided by police all across Europe — would they act the same? If this was something physical, something you could see, would the response still be this passive? 

And honestly, spyware is comparable to a weapon. We’re basically letting companies set up shop in Europe to produce and sell weapons — without any oversight. And if you’ve spoken to any of the victims, or looked at the impact this has had, it’s clear how serious it is. 

Yes, there are the human rights violations — privacy, freedom of expression, freedom of assembly, and so on. But there’s also real psychological harm. We’re seeing victims across Europe suffer from PTSD, anxiety, all kinds of mental health issues. This level of state intrusion into someone’s private life can not only stop them from engaging in political action — it can destroy their sense of safety altogether. 

And no one has paid the price for that. So yeah — I’d say that’s a huge problem. 

00:30:28 Domen Savič / Citizen D 

And one final — or pre-final — question for our conversation is about the media framing of this issue, right? Journalists are one of the most important — or at least one of the most common — targets for spyware developers or users. 

Do you think the way the media frames the spyware issue helps… or maybe doesn’t help… the public understand how serious it really is? Because, the way I see it, you only really hear about it when something happens — in a specific country, or to a specific group, like journalists. 

But there’s no real broader connection being made — no ongoing conversation that ties this to state representatives, Ministries of Interior, or other institutions, right? 

So, do you think the media framing of this issue actually helps keep it alive and unaddressed — especially from a human rights point of view? 

00:31:55 Aljoša Ajanović Andelić  / EDRi 

I’d say it probably depends on the country. The role of journalists in this issue has definitely been relevant — in some cases, they’ve been the ones who actually uncovered the scandals. So sometimes it’s not even companies like WhatsApp or Meta who inform the public first — it’s journalists doing the digging and bringing these violations to light. 

So in that sense, I’d say they play a really important role. 

And then, as victims, journalists have also been very active in denouncing cases — especially when other journalists have been targeted. So they’re involved on both sides. 

At the same time, though, we see a classic divide — the same one we see in politics — where media coverage varies depending on the political alignment of the outlet. So, depending on where they stand ideologically, their opposition to spyware is either stronger or weaker. 

In countries I’m more familiar with — like Spain or Poland, for example — journalists have definitely helped keep the issue alive. Politically, spyware doesn’t get much sustained attention — there’s always noise when a scandal breaks, you get the headlines, some debate… but once that first wave passes, it’s often journalists who keep following the story and bringing it back to the surface. 

So in some cases, yeah, we’ve seen good practices. 

In other cases, the seriousness of it — the scale of the harm — has been downplayed by certain kinds of coverage. And that might be the most damaging effect: when journalism minimizes the threat or doesn’t make the public fully aware that this is a serious human rights issue. 

So maybe what’s missing is more personal coverage — stories about the victims, the real consequences in their lives. That kind of reporting could help people understand just how deep the impact is, and hopefully get more attention on the issue. 

But again, it really depends — on the country, and also the type of media outlet. Because, as usual, the big outlets owned by conservative groups tend to be more state-friendly. Their coverage often leans toward empowering law enforcement and justifying these kinds of surveillance practices. 

On the other hand, you’ve got independent investigative outlets — the ones funded by readers or private donors — and they’ve been really strong, really vocal against surveillance in general and spyware vendors in particular. 

So yeah, I’d say there’s a bit of both. 

00:34:58 Domen Savič / Citizen D 

I’m not even going to dive into the topic of personal responsibility or what individuals can do to protect themselves from spyware. But let’s wrap this up a little differently. What do you think is going to happen in this whole constellation — the EU Commission and the EU Parliament, in relation to spyware, but also everything that’s going on globally? 

You’ve got the Israeli bombing of Palestine, the Russian attacks in Ukraine… you’ve got these actors who are some of the biggest developers of spyware, but at the same time, they’re doing things that clearly don’t align with the European Union’s values. 

So, what do you think will happen? What do you see in the next 2, 3, maybe 4 years in the spyware landscape, and with the EU’s approach to it? 

00:36:10 Aljoša Ajanović Andelić  / EDRi 

Well, this last mandate started off really poorly. In the past year, since it began, we’ve seen the EU and the new Commission really falling into the trap of surrendering to US pressure and big tech interests in some areas. 

I want to believe that the EU will realize the only way to stay competitive globally is by holding on to what made it unique — its commitment to human rights. For example, there are some regulations in the digital rights space now, like the DMADSA Act, which, when we look at where the world is heading, could be seen as really innovative. These are wins for the human rights or digital rights community because they’re putting some limits on these big tech giants, while the US is going in the opposite direction. 

So, I think if the EU recognizes that its commitment to human rights is actually its strength, not its weakness (as it’s been framed in this first year of the mandate), then the natural next step would be to start taking these issues seriously. First, they need to release the communication we’ve been waiting for — it’s been blocked for over two years now, and we don’t even know why. This communication should analyze how spyware use interacts with EU law. 

Then, I’d say the EU could go further. It could tackle the biggest threat we face now: the commercial spyware market, which is thriving and fueling a market for vulnerabilities that puts us all in danger. If I were in the Commission’s cabinet, I’d say the first priority should be stopping the EU from becoming a hub for these companies. It’s self-destructive because it puts all of Europe, even its national security, at risk. 

My second priority would be ensuring that victims get remedies. We know of victims in at least seven countries in Europe, and no one has found any solution for them. The EU has the mandate and the competence to do something about this at any time, so it’s unacceptable that they haven’t. I won’t say they’ll do it, but they should. If the EU wants to maintain its relevance globally, the only way is to stick to what made it strong — protecting human rights, collective rights, and being innovative in defending those rights, not deregulating them. 

If they keep handing over tech regulation to the oligarchs, they’ll be playing into the US’s hands. The US will always be the biggest player in that field. Strategically, it’s a bad decision. So, yeah, that’s how I see the EU’s path forward. 

00:39:45 Domen Savič / Citizen D 

Excellent. Thank you, Alyosha, for dropping by. And thanks to you, listener, for tuning in to this episode of Podcast Citizen D. We’ll be back next month. Best of luck with your work in this field and everything you’re doing in the digital rights space. 

Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!

Z Žano Erznožnik, novinarko centra za preiskovalno novinarstvo Oštro in urednica projekta Razkrinkavanje.si, kjer že več let sistematično preverjajo dejstva in spremljajo dezinformacije v slovenskem javnem prostoru, smo se pogovarjali o problemu preverjanja dejstev v svetu dezinformacij, vlogi novinarstva v družbi in vplivu družabnega spleta.

O politični komponenti dezinformacij, domači in tuji dezinformacijski sceni in kreativnosti, pa tudi o temu, koga Meta ni dovolila preverjati glede lažnivosti izjav in kaj je to pomenilo za dejanski učinek programa.

Za konec pa še o tem, kaj je danes na področju dezinformacij največji problem in česa se lahko na tem področju nadejamo v prihodnosti.

Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!

We are talking to Erik Tuchtfeld, the co-chair of D64 – Center for Digital Progress and the co-author of the “Call for a digital firewall against fascism“, trying to find out the solution to the fascisization of technology and tech services.

We touch upon the issue of big intermediaries, the different policies regarding tech regulation in the US and EU, the role of different interested parties, from the tech industry, the regulator and finally the end users.

What can we do to prevent the political misuses of big platforms, why tech sector needs to pick a side and why time is running out.

00:00:02 Domen Savič / Citizen D

Welcome everybody, it’s the 13th of March 2025, but you are listening to this episode of Citizen D podcast on the 15th of March same year. We are again, you know, catching the deadline…

This time we are joined by Erik Tuchtfeld, co-chair of the D64 Center for Digital Progress, here to talk about the recent Chaos Computer Club post that calls for digital infrastructure resilient against fascism.

Hello Eric and thanks for dropping by.

00:00:36 Erik Tuchtfeld/ D64

Thanks a lot for the invitation, it’s a pleasure to be here.

00:00:39 Domen Savič / Citizen D

Let’s start with the obvious – what does fascism crave in relations with digital infrastructure?

00:00:47 Erik Tuchtfeld/ D64

So, what they’re craving for is on the one hand opposing any kind of resistance, any kind of opposition, so what they’re looking for is a totalitarian and absolutist form of government, basically, and at the same time, to reach this aim, they need to surveil and monitor society.

Both is necessary for them to achieve their aims and with the fascist movement movements we currently looking at a particular in the United States, but also in in in Europe, and in particular also in Germany, where the right-wing party, the AFD, is also rising.

We can see that right, populist, authoritarian movements are on the rise and that there is increasing connection between private surveillance, capitalist big tech companies and actual states, as actual state governments.

00:01:45 Domen Savič / Citizen D

Hmm, so how did we get here, how did we change enter this field, where we were often told that tech companies or technology in general is apolitical, it doesn’t have anything to do with parties or political movements, it’s there for all it serves the greater good of humanity to this really stark and harsh reality where we can see parties and autocratic government cooperating with big tech companies in order to, as you say, surveil and monitor the general population?

00:02:31 Erik Tuchtfeld/ D64

Yes. First of all, I don’t really think that technology was ever neutral, so that is maybe some kind of myth, a myth which is very popular also in the tech scene, the civil society engaged in questions of digitalization…. But technology is never neutral, technology can be used for certain purposes and the way technology is, normally foster certain goals and impedes others.

And that’s fine, I think, because there are certain things we do not want to have in the Internet, and this can also be something where like bipartisan issues and many people agree on, this could be something, for example, spam like people across the political spectrum agree that there should be less spam, or that the distribution of material of child sexual abuse should be impeded.

Where the discussion begins then, is how to do this and for what other purposes and purposes, which are more political, this technology can be used.

You asked… how did we get here? I think what the EU lacked in the past decades was a will to deal with the stark accumulation of power, the concentration of power in the hands of very few. Talking about big tech is not something we do for five years or so, but since 10 or 20 years more or less, there’s talk about the concentration of power, about surveillance capitalism, but the EU only reacted by reforming them, by trying to tell them that they need to deploy their power in a certain way, but it didn’t take the power as such, the concentration as such.

And I think that led to the situation we’re in right now, we have a concentration of power which has very little precedent in the history of basically humankind… that so few people can control the communication spaces of so many, of effectively billions of people.

Currently, I think we see that these people who are in power, who have so much power in their hands, fear somehow threatened, and they react to this threat by trying to exercise total control and trying to diminish any kind of protest and opposition they are observing on their on the platform.

00:05:20 Domen Savič / Citizen D

Hmm, first follow up question. Why was it so hard for the EU to react or act effectively in this field?

00:05:28 Erik Tuchtfeld/ D64

I mean, I think in the end it was a question of political will and the EU, I think, could have acted and we see that in, in certain areas they have also acted… I mean, they are sanctions against, for example Microsoft, also Facebook/Meta, which go into the billions.

But there was never this very strong assumption that the concentration of power as such might be a problem, it was always about the concrete way these powerful oligarchs or these powerful companies exercise their power.

Do you try to regulate the way they’re exercising their power first with the data protection regulation and also, with some antitrust competition measures they’ve taken against the platforms, but the general assumption was always “OK, the system as such is not the problem, but the way it currently works… if you reform the system to a certain extent, this will fix our problems.”

But there was not the recognition that the system as such is flawed and that additional public sphere, which is oriented toward profit, but is currently the case with private companies and not towards the common good… that this is such a problem and that it needs alternatives, that the concentration of power needs to be attacked, for example, with antitrust measures, that something which maybe could have happened more actively in the past, that also mergers…

I mean, we currently see these products which are used by millions of people, are not developed by the platforms or by the companies which own them at the moment, but were simply bought and these kinds of acquisitions, these kinds of mergers could have been controlled more sharply in the past.

The European Union could have also invested more in building alternatives, alternatives to the big tech companies, which are all based in the United States, which are all creatures of Silicon Valley and Silicon Valley’s ideology, apart from Tiktok, something which is also coming from China, but still mainly these companies are based in the US, and in particular, not in Europe, they’re not inventions inheriting European values.

00:07:52 Domen Savič / Citizen D

This is an often-repeated question or a comparison between the US and the EU in terms of innovating in the digital economy or the Information Society, right? We usually say or there’s the general perception that the US is the cradle of innovation while the EU is the cradle of regulation, right?

Would you say that is factual, that is true, or is there some other… are there some other powers in play that the enabled the US to be the cradle of innovation? If it is in fact, and on the on the other hand, that it tasks the EU with regulation?

00:08:48 Erik Tuchtfeld/ D64

I mean it is a very common, it is repeated by hundreds of scholars and politicians, it’s very popular and I guess there is some truth in it, but still, I don’t think that it is that easy.

First of all, I’m not really sure if the United States are so good at innovation, there’s a lot of innovation also going on in the European Union and we have a very active open source and civil society scene here in in Europe as well, maybe even more active in certain parts.

But what American companies are very good at is scaling, getting big really, really quick, and then also maybe not caring so much about the problems they’re causing for others in this process of scaling and this process of getting bigger and bigger.

I mean this is permanently coined by the term “move fast and break things” and maybe exactly that is something which is not possible in the European Union because there is more regulation and there is more supervision over rules than in the United States.

But that’s not, that’s not a bug, it’s a feature, right? We don’t want certain practices, we don’t want companies to break things and I mean break things somehow sounds harmless, but when we talk about Facebook, for example, there is this concrete evidence that the way Facebook moderated content led to a genocide. So that’s not simply breaking things, but it’s harming and being responsible for the death of many, many people.

And EU regulation, maybe that is the hypothesis I would like to phrase, impedes this kind of innovation, this kind of scaling, but it’s a scaling which does not really benefit humankind and, I am being a bit dramatic, I don’t think that something is missing when we don’t have this kind of scaling, but the question is how do we protect ourselves then against the scale factors of other companies of us American companies?

Because with the scaling there comes a lot of interesting side effects which makes it more difficult then to switch to other alternatives. There’s for example the network effect that you want to go there, where many other people are because there’s the interesting content, they are locked in lock in effects so that it’s really difficult to leave certain platforms to follow the content from other platforms because they’re normally not based on general standards, they’re not interoperable…

So, all of that, all of those are then certain measures, measures which were consciously taken by the platforms to hinder competition and to make it more difficult for alternatives which are not oriented toward profit to enter the market.

00:11:57 Domen Savič / Citizen D

I want to stay at the genesis phase of this current situation that we find ourselves in…  So, so would you say that the political discourse or political agreement with these, as you said, mostly U.S. companies that were growing like a virus on a global scale sort of suited the decision makers in that these companies were at one point providing a service that maybe, not exactly, solved a lot of issues, but was very fitting also, for the decision-makers to sort of reach out to the public, to sort of communicate with them, to sort of have this benefit of this, pulse measurements of the public opinion through social media and other digital inventions?

And would you say that was the reason the regulators, left them alone for so long and did not address these issues that that are or that were popping up for the better part of the last ten to fifteen years?

00:13:21 Erik Tuchtfeld/ D64

I think that might be a relevant aspect… I mean social platforms and also US, American-owned social platforms are not bad as such, I would argue they’re also connecting people and are the basis of platforms where many social movements, liberal social movements, movements which are critical of power, are forming and have formed.

So, there are relevant communication space for many, many people, and they’re also relevant communication space for politicians because they can reach citizens, they interact with them, they can discuss matters with them and they, as you said, they get a grip on the pulse of society.

There are a lot of flaws, but it’s always important to me to emphasize this aspect that not everything is bad and what we need to look for is alternatives and not simply shutting the system down as such, because some argue at the moment for some kind of service shutdowns, there are particular countries which have even introduced Internet shutdowns as such, and I think there are concrete dangers for fundamental freedoms and fundamental rights, in particular freedom to expression, freedom to impart information, but also many other fundamental rights.

I think you’re right, that it is interesting that is and was interesting for politicians as a tool to communicate and as a tool to interact with society, and at the same time also from a regulatory perspective, it is, to a certain extent, it’s convenient to have a single point of contact, a single point of access, when you want to enforce certain rules.

So, in a decentralized Internet where there are many different institutions, many different companies, organizations all providing their own public spaces with their own rules, maybe even when the regulator finds that something is needs to be stopped a certain behavior, or that a certain measure needs to be enacted, enforcement becomes very, very complicated because you have so many different actors you need to address, and it’s just it’s just difficult and now when you don’t want to have, for example, a certain application to be provided to smartphones, there are only two companies you need to go to and when they don’t comply, you can issue fines and sanctions.

So, it makes regulation and the enforcement of rules in certain ways substantially easier when there is a concentration of power, and so there is only a very limited amount of counterparts you have to talk with when you want to enforce your laws online.

00:16:19 Domen Savič / Citizen D

Because looking back at the role of the Internet or the social networks, these big, mostly US-based companies in different say social movements or demonstration or protests… There’s always this “yes and no” position in terms of how useful these tools are in generating or keeping the social movements alive?

So, for example, looking way back, you know, the Arab Spring was one of the first maybe or one of the most important social movements or protests or waves of protests in the Middle East, where everybody in the beginning were saying, you know, social networks, digital platforms were, you know, harbingers of democratization in in those societies, right, Egypt and neighboring countries.

Then after a while, these claims were scaled back in terms of, “Oh, no, no, no, this wasn’t as big as it was reported before, the networks played some role, but people would still find a way if social media wasn’t present in the society”, and then you had, let’s say in the 21st century, you had the wave of Russian propaganda and all of the sudden every aspect of social networks, of digital platforms that was before used as an argument for the platforms was now used as a critique of platform – they’re brainwashing people, they’re spreading disinformation and so on.

So, in terms of regulating this field, why do you think the regulators were so slow to catch up to the fact that, “OK, if we can spread the right message and are effective at doing that, somebody else could spread “the wrong message” and also be effective at that.

So, this, as I said before, this mythical neutral technology versus technology as a tool to effectively spread a particular political message that is fitting or isn’t fitting with the ruling structures?

00:19:24 Erik Tuchtfeld/ D64

I mean it’s a very good and very difficult question because it’s not only about what would be the best regulation, but it’s also about what are the motives of past legislators to not regulate so much.

Before the European Union became very active, in particular at the beginning of the 2020s, but also before with the GDPR, which was kind of the starting point of digital regulation of the new wave of digital regulation in Europe, my impression is, which is also still valid when it comes to current regulation like the Digital Service Act, that there is a certain hesitancy because we don’t really know what we want or legislators don’t know what they want, because there’s not a clear image of how good digital public space, where people meet and convene and can exercise their protest and can communicate political issues and stuff like that…

There’s not a clear answer how does it look like? There might be certain shared values on how it should not look like, what should not be there, like constant surveillance, which is currently the case.

Maybe the EU should have put more emphasis on what should not be there and then try to prohibit this completely. For example, as I said, the constant surveillance on digital platforms, the personal advertisement and stuff like that. They’re doing that now but slowly and obviously, also with a lot of opposition from industry.

So, I think what the past has shown is that social media can be a force for the good, that there are good things happening on social media, you’ve talked about the Arab Spring, but we can also talk about Black Lives Matter movement, we can talk about Fridays for future and the climate movements.

As such, I think many in particular movements which are somehow established with younger people, youth movements, they have certain roots also in social media and I’m not saying that without social media we had no protest or we had no movements. Obviously, there were social movements before and there will also be social movements when this kind of social media does not exist anymore, but we at least facilitated by the way social media works.

Also, apart from these big movements, social media companies or social media platforms are spaces where marginalized people meet, where people who are maybe living in the countryside and who are not aware of any other person who shares their certain characteristics and who feel really lonely, they can meet friends online and they build a community and feel to be part of a community and be empowered through these tools.

I don’t know about you, but also many in the civil society, many people who are organized in civil society and who are very critical of big tech are still using their platforms because certain aspects connecting people works and that’s I think something what also regulators have acknowledged in the past and they have seen there is something good and the platforms are doing.

At the same time, they’ve also seen the problems. But there was a lack of a vision of how it should look like when it shouldn’t look like as it is and I think this lack of vision is something which is still a problem. I don’t have the impression that the European Union and also the politicians at the top of the European Union, have a clear vision of a digital public space which is committed to the public good. They just look at this at the current commercial public space and say “That’s not the best we need to improve this, there needs to be a bit less hate speech, there needs to be a little less disinformation, but we don’t really know how it should look like, which is really a different model and which is an alternative”.

00:23:52 Domen Savič / Citizen D

If I can play a devil’s advocate a little bit to what you just said… so, if you think about the role of social media as a force for good, as platforms organizing, bringing people together and at the same time you wonder from a position of a digital activist… How much of this usage was actually the necessary evil?

If you talk about or if you look at the role of the general mainstream mass media, right… in terms of telling us what’s going on in the world, contextualizing the happenings around the world, focusing on different perspectives and let’s say social groups or minorities… you can clearly see that in the last few decades, they were basically asleep at the wheel, because of many problems from advertising to the whole profit model of mass media and similar stuff.

And then you had on the other side, you had this upcoming new actor in the field that offered as networks usually do, free, nice, happy, “just be yourself” type of agenda, but at the same time, you know, looking at it from today’s perspective, we were basically sheep brought to the slaughter because, you know the whole system was based in in surveillance capitalism, right? It was based on our personal data… And this was something that was always built with an intention of bringing together huge swaths of personal data and using it for purposes of advertising, of state security and stuff like that.

So, thinking about alternatives and thinking about the role of our, let’s say, functioning society… what are your thoughts on that? Where do we go from here? It’s 2025, you have uprisings of far-right movements literally all across the globe, you have these digital platforms playing favors with them. What’s next?

00:26:26 Erik Tuchtfeld/ D64

Yeah, I think… the law as it stands at the moment won’t save us, so there are many currently asking for very strict enforcement of the Digital Services Act against the US platforms which are now so aligned with the right-wing government in the United States, and I don’t think that that will be enough. It’s necessary, but it’s not enough.

What we need instead is, I think, a more active approach towards these platforms where the concentration of power as such is a problem. So, I would say, we really need to break up big tech, there shouldn’t be platforms which are that powerful. At the same time, this is not very realistic, because I don’t think the EU wants to escalate the conflict with the United States further at the moment.

What might be a possibility is a more active interoperability obligations for the platforms so that the effects so which lock users in at the moment, so they have to stay there because they don’t get the content anywhere else…that these effects are diminished.

A reform like that that could be implemented, for example in Digital services Act or the Digital Market Act, might be more realistic than a complete break up and I think what we need is alternatives and we have, I mean some alternatives like fediverse which conceptually I would say are really, really convincing.

I mean when it comes to ditch the public spaces committed for the public good, the way the fediverse works and also its most prominent service, Mastodon. I find it really convincing; there’s a lot of possibilities to experiment with certain governance structures to legitimize the rules which are governing these spaces by the users which are affected by the rules, which is, I think, a super nice and kind of a novel idea for internet communities, at least when it comes to social media communities.

So, I think there’s a lot of potential, but at the same time we can clearly observe that it doesn’t really fly, so there’s so much going on and still it’s not like all of our neighbors, at least not my neighbors here are using it… if at all they use any alternative, they use bluesky, which is of course… I mean it’s not-for-profit company, if I’m informed correctly and theoretically, it could also be decentralized, but it’s very difficult so it’s better, but it’s not really a super good alternative.

So, they preferably using this and why is that the case? I actually think mainly because of algorithmic creation, because, I mean, Mastodon prominently does not use any kind of algorithm apart from chronological sorting and I think that people want to have some creation over the content of social media platforms, they simply do not want to be online all the time to see what has been posted right now. And then I see it in my feed and when I come online at the end of the day, I don’t see any more what was posted.

In the beginning, I would say, one of the weaknesses which needs to be improved in the very concrete design of Mastodon and to enable such developments, I mean that’s not a very concrete proposal, maybe that’s also not the fix for everything, but there are also other flaws in the software as it is at the moment, which hinders some people from using it, and I think the software could be more accessible.

What we need for that I think is also funding, so there needs to be a European idea of how social media platforms should look like and then there needs to be public funding when we say it should not be profit oriented. When we say a society say, it should not be profit oriented but committed to the public good.

And I think with this funding, there comes more development, the tool becomes more attractive and then it attracts more users, more interesting content, which will then hopefully be a vicious circle in the best sense and to make the alternatives more prominent than the current established platforms.

What I don’t think the European Union or any other big stakeholder in this field should try, is simply to build up the next Facebook, the next Instagram, the next Twitter/X and try to copy and create some kind of European giant, just a European centralized platform which at a certain point might have the same issues when it comes to the concentration of power as we currently face towards the United States owned companies, but it needs to be an alternative, where decentralisation of power and autonomy in the sense of self-government of the users is somehow embedded at its core.

00:31:42 Domen Savič / Citizen D

So again, playing devil’s advocate, you focused on the platforms themselves, on the way they were built, on the way they used algorithmic content moderation to grow, gain influence… but what about the broad situation in a society, right?

Do you think part of the reason for their growth was also a sort of social optimism that was present during their period of expansion of growth, there was talk about technology saving  the world, there was talk of digital literacy, there was  digitalization happening all over the place… and now in 2025, we are facing, I’m not going to say completely, but very different reality in terms of what the tech is actually doing, how is it solving or addressing some issues and do you think that plays a factor that even if a new platform or a new way of communication literally drops from the sky tomorrow, the same thing that happened in the early 2000s will happen again?

00:33:23 Erik Tuchtfeld/ D64

Yeah, I mean that’s a fair point… I think that infrastructure which is organized in itself against… which has some mechanisms which try to make it more difficult to abuse power, which try to make it more difficult to concentrate power, that would be a feature which was not in mind 20 years ago, because you’re completely right the way the big tech platforms grew, it was very much founded on this idea of techno optimism that technology would somehow save the world.

This is also something we can currently observe when in conversations about artificial intelligence that this will somehow just like magic save the world and solve all our problems.

And I think what we have to acknowledge is that those are social technical systems, they’re embedded in society and society has its flaws and problems and there will also be reflected in the technology at the same time, we are aware of the last 20 to 25 years.

So, I hope that when we create something new, when we design something new, we are a bit smarter than we were 20 years ago and that we would try to at least to make it more difficult for the abuses we are currently experiencing to happen.

Again, so that gives me some optimism, that we’ve learned something from the past and that when we create something new, we our new designs are informed by what has happened in the past.

But at the same time, I don’t think that the new social media and alternative social media for the public good will not in itself solely fix the rise of authoritarian movements and I also think that it’s not only social media which is important or which is responsible for the rise of these right-wing authoritarian movements.

We can currently also observe sometimes in the discussion that one has the feeling if only big tech wasn’t there then all problems would be solved solved and Donald Trump would definitely not be president and the AfD in Germany would not have 20% or maybe would not even be in Parliament… I don’t think it’s that simple, I think it’s a very complex interdependence and I think that social media platforms have this certain share in the current rise also of authoritarian movements, but I think it’s also about bigger problems with society and we somehow… we all do this as activists because we are, I mean at least my understanding, we are political activists, fighting for more progressive future and our current field of application is digitalization. But we’re part of broader movements who are fighting for different and a more open society and that is something we… I mean we don’t need to fix just social media, but we need to work for a transformation of society which is more just, more open and also takes into the account the limitation of growth and climate justice and many different aspects.

I think that that is important to keep in mind that there are certain flaws in social media we can and should fix, but that this won’t be enough to fix society as a whole.

00:37:07 Domen Savič / Citizen D

And since we’re slowly wrapping up, I just want to return to the original reason for this episode, the Chaos Computer Club post… So who did you have in mind when or while you were collaborating, writing this call for a tech resilient against fascism cravings.

Was it focused on the decision makers, was it aimed at the tech sector, the end-users, the activists? Who do you wish would answer it?

00:37:54 Erik Tuchtfeld/ D64

So, the letter or the post which was published in the English version by the Case Computer Club in the German version also on our website at D64, it’s a joint effort of more than 20 civil society organizations in Germany and the very current trigger to write such a request letter are the coalition negotiations in Germany.

A new government is forming in Germany and we try to influence the way they’re doing tech policy… It’s directed towards decision makers at a political level and it has three main sections.

First is against surveillance, one is emphasising that protection, security for all people, in particular marginalized people, is necessary and the third one is on fostering democracy and democratic participation in the digital space. The very current trigger are these negotiations and they’re addressed towards policymakers in Germany.

But there is a reason why we published it in English as well, because we think that that the concrete proposals, we make are not only something for German policymakers, but for policymakers and also civil society organizations and even end-users around the world.

We think that surveillance as such is a constant problem, and I think it’s something which needs to be fixed on the policy level and which should not be individualized. That being said, there is some kind of resistance, which is also possible as an individual or as a civil society organization, the use of alternative tools which are not based in the US, the use of alternative platforms which are not based in the US but which are maybe self-governed, we’ve already talked about Mastodon.

Understanding the themes of this letter; protection and security for all, in particular marginalized people, democracy and democratic self-governance in the digital space and the commitment against surveillance is something which cannot only be fulfilled by policymakers, but something all of us can try to improve and can try to support in our daily lives, also as tech users.

00:40:28 Domen Savič / Citizen D

This is this is lovely but I just have one devil’s advocate question for the conclusion. Do you think all of this makes business sense? So do you think this can be a convincing argument for the industry that was built around for the better part of the last two decades on the exact opposite of what you just described?

00:41:01 Erik Tuchtfeld/ D64

I mean, I don’t think it’s really a business case we are describing in this letter but at the same time, the industry also needs to decide on what side of history they want to be, and currently with the alignment of big tech companies in the United States, with a fascist and authoritarian government, I think that those Members of industry which are not yet aligned with the right wing movement, they have to know that there is no neutrality in this situation.

So when they want to protect themselves from a government which seems to know no limits anymore, they need to change their products in a way that they are not a danger to users anymore, they simply must not collect any data which are not completely and absolutely necessary for their services, and I think being aware of this responsibility towards society, also as a company and also as industry is something which might in at least some cases, might respond to that, I have some hope.

I don’t think that that Meta will change its business model, I don’t think they Elon will read our letter and then will realize he should really step down and somebody else should take over. But I think that also in industry, that industry is not just one block which is completely aligned and which is homogeneous, but that they are diverse voices and with this call, we are trying to address those voices which are maybe currently in quiet opposition towards what is happening in the US at the moment and that they need to become more active and that they have a responsibility towards their users to protect them by making sure, for example, not to connect collect any data which can be abused by state authorities.

00:43:08 Domen Savič / Citizen D

This was great, thank you so much Erik, for dropping by, for, for sharing your thoughts and wisdoms, thank you, dear listener, for listening to yet another Citizen D podcast episode, where we publish an episode every month, thank you all and see you soon.

Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!

We sat down with Joe McNamee, senior policy expert at the EU DisinfoLab, who was previously leading the European Digital Rights network.

The topic of the discussion? Everything from digital rights to fighting disinformation, the current EU political climate and more.

How does transparency come in to play when addressing the digital rights and countering disinformation, why the current EU situation does not bode well for guardrails in both fields and what can we do about it?

00:00:02 Domen Savič / Državljan D

Welcome, it’s the 11th of February 2025, but you’re listening to this episode of Citizen D Podcast on the 15th of February same year. With us today is Joe McNamee, a senior policy expert at the EU Disinfo lab who’s been working on topics related to Internet regulation for over 20 years.

From 2009 to 2018, he led European digital rights, so-called Adri, the Association of Digital civil Rights organizations in Europe, working on major topics such as adoption of the general Data Protection Regulation and the Copyright Directive. So welcome Joe, welcome to the show.

00:00:43 Joe McNamee

Thank you.

00:00:45 Domen Savič / Državljan D

Let’s start at the beginning… there’s tons of things happening in the disinfo area of EU policy and practice as well, but I want to start with your previous or pre-previous employer; European Digital Rights Association.

How did this field of digital rights change in the last, let’s say, 10 years, the so-called digital decade. What were some of the challenges that were, let’s say, present at the beginning and what were some of the solutions we saw as the as the as the decade wrapped up?

00:01:30 Joe McNamee

Well, I’d say the problems are easier to define than the solutions because the problems are inspired by things that seem to last forever, and solutions change as technology changes. So, I think over the last 10 years, as you mentioned, the general Data Protection Regulation getting across the finish line was already a success, and it being a reasonably solid instrument was also a success.

We had the adoption of the Digital Services Act and the Digital Markets Act on a European level which served to strengthen the role of the of individuals in society in the digital society, we had things like the latest revision of the Audiovisual Media Services Directive, which is more important than people think in regulation.

What we see in here in the audial space, particularly with video streaming services, both based on profiling like YouTube and paid for like Netflix… So, there’s the one thing I think that is a fundamental change is that I remember in 2014, two 1015, there was a lot of objections to the notion that there were digital rights.

Whereas now I think the question is answered that the rights valued in the pre-Internet age have a different character and a different value in the in the digital age, in 2025 and we have to think about what it means to transfer those rights and to protect those rights in the digital environment compared with the more analogue life that we had prior to 2015.

00:03:50 Domen Savič / Državljan D

This was actually my follow up… so how do you see or how did you see the change in, let’s say, civic organizations or the general public, the change of perception of this field.

So, I remember like back in was it 2011, 2012, one of my first, let’s say campaigns was against the ACTA Treaty and back then it was really hard to sort of broaden this this issue of treaties of surveillance of, you know, monitoring, monitoring the different digital past. It was very hard to translate this into a language that the general public would understand, right… as we progressed it seemed in a way that the public is getting more in attuned to these issues, to these values. There’s a bigger connection between, as you said, human rights and let’s say digital rights.

But again, looking at the past 10 years, how did you see this area changing or adapting to the to the new reality?

00:05:04 Joe McNamee

I think it’s a question of people and society being better able to map what they consider to be unquestionable values of society like democracy, like privacy, like freedom of expression, mapping that onto the digital society.

And I think ACTA… it was perhaps one of the first examples of a certain part of society going “We can’t just leave this to arbitrariness…”, we have to have better control of our freedom of communication online and since then, we’ve had different challenges, different opportunities, like the update of the Data Protection Directive which became the GDPR, where there’s a lot more understanding among policy makers among politicians than there would have been two years before, or even or three years before.

I remember when the GDPR was proposed first by the Commission, there was… It was met with sort of a degree of bafflement, and the degree of misunderstanding and deliberate misunderstanding to a certain extent. So, I think it’s society becoming more in tune with old rights in a new digital form, I think that’s really the big change over the past 10 years.

00:06:46 Domen Savič / Državljan D

OK. And what were some of the drivers of this attunement?

00:06:54 Joe McNamee

I think, it came to something like data protection, I think it became more obvious just how much data was being generated about each individual and what could be extrapolated by computers … if we imagine the foundational work for the GDPR was done in the early 1990s and the number of computers that existed and their computing power was fairly minimal and we were talking about protecting personal data that was largely on sheets of paper. And I think in the course of the… between 2010 and 2020, it became more embedded in society that it’s a whole different level of magnitude; personal data being protected now with huge amounts of data being collected and data that can be compared with each other.

I wrote a paper for EDRi, back in the day, where I talked about data that had no parents. So, data that was generated by comparing two other pieces of data. If you’re this, and if you’re that, then chances are you will be… If you’re A and you’re B, chances are you’ll be C and the awareness of that change in what those that hold the data and those that are allowed to process the data can do with it was finally understood in that period. And I think even in 2014, I think it was when the initial conversation started, society wasn’t there yet, hadn’t quite understood and I think now you know that your data can be used in this way and I think that is a fundamental shift that allowed good things to happen, even if implementation of the GDPR is sometimes disappointing.

00:09:33 Domen Savič / Državljan D

And how do you see the difference between, let’s say for quite some time the, let’s say the good guys were the big intermediaries, right? Facebook and other US-based companies and the bad actors were these shady companies that nobody really knew about.

As the time moved forward, you had this… I felt like a change between the perception of these big intermediaries that weren’t all good and at the same time the issue of foreign states came into play, right? You had the US, let’s say China, Russia in in the last couple of years… so do you think this change from commercial to geopolitical sense of digital rights helped the progression of this field?

00:10:49 Joe McNamee

I’m not sure there was ever a possibility to make a complete distinction between state and private because we saw… most simply were in the Data Retention Directive that showed, once data is stored and once data is processed it will be kidnapped for and repurposed for state reasons… So, I think one leads to the other rather than one being different from the other.

Of course, China went off in in one, the US, with its big tech base, went off in another direction and is going off in a new direction now, but I wouldn’t see them necessarily as a distinct phenomenon.

00:11:53 Domen Savič / Državljan D

I mean, sure, they’re connected. But at the same time, I get the feeling that, when we were discussing, let’s say, the GDPR, the main actors that the data needed to be protected against were the big intermediaries.

Right now, you have… after or maybe even during or just before COVID, you had this shift to a more geopolitical sense, where companies were maybe replaced by country names or regime names in a way, right.

So, you were talking or we were all talking about, you know, EU digital sovereignty in regards to both China and the US, you had this whole push towards more EU-centric development of private, but also public sector in this regard.

And on the other side you, you also had these quarrels between the West and the east, right, so you had the US countering the push of let’s say, the Chinese, then you have Russian companies in in this regard.

So, I was trying to lead into the next topic on the agenda, which was: are our digital rights, something that pertains mostly to the consumer or are our digital rights, something that pertains more to, let’s say, a citizen?

00:13:50 Joe McNamee

I think… if something is a value, then it’s a value regardless of time to a point. So, the rights that we have are rights that we see as values that underpin our society, even if the challenges to those values, whether it’s privacy or freedom of expression or whatever, the right to life… The things that need to be protected so it doesn’t… it’s easier to explain that in one year by explaining the threat from big tech and another year, it’s easier to explain because from a threat from government perspective, but the actual underlying value that our society holds dear is timeless to a point.

For example, there was the profiling case of the Dutch Social Security Office,  I can’t remember the what it’s called anymore, that did some illegal profiling of individuals in order to assess whether or not their applications were appropriate and it became easier to explain to Dutch people the danger of state profiling, because there was an example that was fresh in their minds as regards what had to be avoided and what the underlying values of society as reflected in human rights law are required to be protected, whereas if you if you go to a Portuguese person or a Spanish person trying to explain the same thing you need, you can’t explain in the same way because the society’s experience is different.

So, it… I don’t think it’s better to focus on the value that the right represents rather than it necessarily being something that needs to be protected, either from private business or from the from the state… it changes and it will continue to change.

00:16:38 Domen Savič / Državljan D

But do you think the approach to protecting these values is the same or… are there some similarities between arguing for privacy against Google and big intermediaries or against a democratic or somewhat democratic state?

00:17:04 Joe McNamee

I think… nothing is so good that it can’t be done badly and try as we try as we might, we always find ways of doing good things in bad ways. So, if you say a state democratic state by definition doesn’t need the accountability of data rules, then that is that democracy will be eroded by the lack of accountability.

So, it’s a very dangerous and counterproductive thing to say, we can trust these people, but not those people, because that’s, that’s where the erosion of morals and values happens. No government structure, no private structure is good enough without accountability to withstand the temptation to take shortcuts.

00:18:18 Domen Savič / Državljan D

Looking at the EU level, but also how does this regulation and these protective frameworks transfer between the Brussels level, the EU level to the to the Member States?Do you see this as an effective system of translation in implementation or is this something that is actually even more important than the regulation itself but is then sort of diluted or spread so far apart that it’s very hard to follow it to the actual enactment of all these EU regulation in practice?

00:19:12 Joe McNamee

And that’s a question that I would be very rich if I could answer it effectively. The GDPR relies on national data protection authorities and they vary quite a lot in in quality… and this is traditionally the way the EU did business and traditionally works quite well, if the issue is within the borders of a country.

So, if you take telecom regulation, for example… some countries implemented well, some countries implemented badly. The countries that implemented well, have fast and innovative companies with low prices, the countries that that don’t have higher prices, less competition, less quality telephone and Internet connections.

That’s not good, but ultimately, it’s a problem that is or it’s either a problem or solution that is chosen by that country for that country and so at a given moment, before the GDPR was thought of it made a degree of sense.

So, when the GDPR came along, they followed the same approaches, the 1995 data protection directive and they said, “OK, we’ll leave it to the Member States to implement”, for companies that are based in those countries.

But what happened was that that made the traditionally weak Irish data protection authority responsible for Google, Facebook and every other big tech that wanted to establish a base in in Europe and an underfunded unsupported authority became responsible for vastly more than it could ever have been able to manage and that has led to significant problems for the protection of personal data for individuals in the EU and beyond.

And you saw that in the in the discussions around the Digital Services Act, for example where one of the things that was repeated often over and over again in the in the political discussions was we can’t have what happened to the GDPR happen to the Digital Services Act and there was a push from the center and the right of the European Parliament to say we can’t have a new agency, ideologically, we just we can’t defend new agency.

So, the compromise that was reached was that the European Commission would be the regulator for big tech, taking that away from the risk that it might fall to Ireland again allegedly and the national authorities would be responsible for everything else on the national level.

So, there’s been an evolution of the thinking of the of the EU from a time of pre-networked age where everything could be left to national choices that would have national consequences to a network page where we have networked data, which and networked freedom of expression and so on, that requires a higher authority… which for good or evil, is the Commission in this case.

00:23:34 Domen Savič / Državljan D

And you know, looking at or comparing the field that you’re working in now, so the disinformation or the tackling of the spread of this information and the digital rights, so your previous job… What are some of the similarities and what are some of the differences?

Because in a way I thought about this really weird connection where one could argue that on one side the digital rights were more about freedom of expression and the uninhibited transfer of information across the Internet, while the disinformation field is slightly different in a way that not everything that moves freely is actually good.

00:24:30 Joe McNamee

Well, that goes back to my frequently repeated mantra that everything can be done badly if you try hard enough and defending digital rights and the basis that nothing has any limits is as wrong as saying that everything needs to be restricted by default. The truth lies somewhere in the middle.

If you leave it to big tech to control what you see and hear on the basis of reverse censorship of pushing the information that they see as profitable up and pushing everybody else’s freedom of expression down… that’s not defending freedom of expression either. So, a balance needs to be to be achieved where voices are heard and without undue interference by the companies. And in the same way, there are fundamentally anti-democratic forces that are organized in in spreading disinformation which removes the freedom of expression and democratic rights of individuals and groups that need to be addressed without necessarily preventing anybody from any human being from speaking.

I thought that there was when I before I started this job, that there might be a friction between my strong belief and freedom of expression that I defended in EDRi and my strong belief in democratic rights, which I defend in my current job, but actually there isn’t because you have to both can be defended at the same time and it’s the.

If you look at the pre-censorship of algorithms that are promoting toxic content or foreign information networks, those are both contrary to democratic rights and country to the freedom of expression, rights of individuals and groups.

So, I I’m happy to say with complete certainty that the two go hand in hand, and they’re not contradictory to each other.

00:27:49 Domen Savič / Državljan D

Because I was wondering about that… let’s say these autocratic forces are usually using the freedom of speech protections to sort of further the spread of disinformation, while you have the counter-argument of companies that are, you know in charge of algorithmic content management are sort of, you know, left alone or they’re usually, you know, defended by “Oh, no, no, these are private, you don’t need or you don’t have meddle in them”, right?

So how do we address the issue of at least on paper or in practice or on private companies that are now becoming more and more essential and more and more inescapable, if I can use this term in a modern communication sphere.

00:28:59 Joe McNamee

I think we can solve multiple problems with one effort and the effort is to be found in the shortest article in the Digital Services Act, and I think possibly the shortest article in the history of European legislation, which is Article 38, which says that there must be non-profiling based options available in very large online platforms that are providing services.

If you give people the power to build their own feed of information that they that they want to see and people that they want to hear from and get recommendations from those people about other people, they want to hear from rather they want to hear from, you have an increase in media literacy because these because people are able to understand what a menu is and what choosing your based on your tastes is.

And that means that different people would have different feeds and there wouldn’t be the single point of failure at which we have at the moment, which is the monolithic algorithm feed which is designed not for the interests of the user, but for the commercial interests of the platform.

And I think if we could, if we could break that monopoly of information feed in the platforms, we would have platforms that would be serving society in a more transparent way, in a more safe way, because it’s comparatively easy to work out the basics of how an algorithm works and exploit it, it’s quite another thing working out how to overcome an algorithm that is based on the tastes of many million or a billion subscribers?

That’s a personal opinion rather than one of either former employers or my current employer.

00:31:55 Domen Savič / Državljan D

But is that, you know, feasible in… Because then you’re usually left with the question of these platforms, they are running these algorithmic content decision-making solutions on the other hand, you have, the end users and state entities.

But how do you make the whole thing work? So, we’ve met back in 2006? and the one of the first issues even back then was these self-regulatory models do not work when push comes to shove, so we need stronger role of the state impacting this field, right?

And as we moved on, we saw even in the digital decade that the States aren’t really pulling their weight, but you know, compared to always reverting back to self-regulatory models and codes of conduct… there was a lot of the latter and not a lot of the former, right.

So is there… I know it’s $1,000,000 question but is there an effective way to change the practice in this in this regard… to make it more functional and effective, and actually doing what it says it does on paper.

00:33:36 Joe McNamee

Well, I would first of all ask whether it’s possible to make it less functional than it is at the moment because the monolithic algorithms are causing harm and any improvement would likely be a change for the better.

I think one important thing to understand to accept is that self-regulation doesn’t always work, but self-regulation does sometimes work, so the Council of Europe adopted guidelines on best practice for content moderation, for example, and that sets out some very basic principles for what rules need to be followed in order to have a self-regulatory approach that achieves public policy objectives for the good of society.

And we need to move beyond embarrassing levels of naivety of governments when they when they approach this topic. If it’s to the economic disadvantage of a company, to self-regulate in the way that governments want, it will not do it well, it will not do it at all possibly and having a press release and having a photo opportunity saying something different is not going to work.

So, in environments where self-regulation is not going to work, then regulation has to work and the DSA Digital Services Act from the EU tries to strike a balance between flexible, rapid measures that can be taken in a self-regulatory environment together with several articles on transparency in order to ensure the appropriate levels of evidence and data are produced to show when regulatory intervention is needed and necessary and proportionate and rules that apply in all cases.

I think that the architecture, if not necessarily the outcome, is an admirable one and one that potentially could have been done better, I think. I think it’s a model that isn’t a failure by default.

00:36:37 Domen Savič / Državljan D

And how do you see the issue of, let’s say talking or staying on the topic of EU regulation, right? How do you see the issue of these changing decision-making bodies on the EU level? We’ve just got a new European Commission; we got a new European Parliament. They’re now deciding what they will be doing for the next 5 years… How do you see the next five years in the area of digital rights?

00:37:23 Joe McNamee

That’s a very big question… the only thing that we see clearly at the moment is the noise around enforcement of the of the Digital Services Act, the noise around the creation of the Subcommittee on the Democracy Shield, the majorities in the Parliament are quite difficult to build because of the election results. Plus, if anything is possible on a scale similar to the GDPR of the DSA seems quite limited, the Commission is under tremendous political pressure to not do anything that would be perceived as undermining financial security or the economic well-being of the Union.

So, every direction that you look in, there’s a constraint on the Commission as a whole, the strange division of powers that the Commission President has imposed on the Commissioners is a restraint on all of them to act freely in the best interests of the people of the European Union, in my opinion.

And even if there was something strong on any subject that was going to come out of the European Commission, there are forces in the European Parliament now that are partly there because of external forces that don’t want Europe to have effective regulation on anything and therefore building majority to pass legislation that the European Commission will have difficulty proposing is limited as well.

There are significant barriers to there being any major, meaningful policies adopted in the next five years, in my personal opinion.

00:40:01 Domen Savič / Državljan D

So, will there be more bad policies or as you said before, it’s going to be a stalemate where nothing will move at all?

00:40:15 Joe McNamee

I think there will be an effort to deregulate, to move backwards in the name of bureaucratic simplification. And how far that goes, I think that sounds nice, it sounds sellable, “we’re simplifying bureaucracy where we’re getting rid of red tape,” but the red tape was designed for the achievement of a public policy objective that was shared by a majority in the Parliament and by everybody in the Commission.

So, it’s a seductive term with… the least bad consequence is the removal of measures that were designed for the good of the people in Europe and businesses in Europe, in and with the majority in the three institutions.

So, I see more political potential to move backwards, being seduced by simplification and getting rid of bad red tape than I see moving forward with good legislation, good policy development and legislation.

00:41:56 Domen Savič / Državljan D

And if we in our last part of the show, if we move on to the disinformation and compare all of the work that’s been done in the last digital decade on the issues related to digital rights and now try to talk about the current situation in the field of disinformation…

Do you think the next five years of the new European Commission will provide some solutions, some regulatory frameworks that will address the issue of disinformation or is this, with the current powers at play sort of a lost cause or an issue that you know nobody’s going to touch?

00:42:55 Joe McNamee

Ideologically, I don’t like the notion of condemning somebody for a failure that they haven’t failed at yet. I think in principle, the Digital Services Act, Digital Markets Act and GDPR give us all of the tools that we need to within the European context, within the EU context to address to, to make major take major strides in the fight against organized efforts to undermine our ability to choose by undermining our ability to know what our choices are.

It is still a little bit too early to tell if the implementation phase will fail, I hope not. But I think we have a duty to not condemn those who are trying their best to impose, to implement legislation that was adopted for the good of our society, because even because the best way of ensuring legislation will fail is by talking it to death before it has it has had a chance to succeed.

00:44:26 Domen Savič / Državljan D

And staying in the disinfo area, what would be some of the key issues or the most pertaining problems that the EU is facing in the current situation?

00:44:48 Joe McNamee

This is again more of a personal opinion, but I think disinformation, the economics of it, of how disinformation is funded and shared is central, if we start with the algorithmic choices made on social media that promote certain content and demote other content, is a problem that is inherent to any system that is reliant on an algorithm which is not in the interest of the user and not is not in the interest of quality, but in the interests of driving traffic for the purpose of generating data.

And that’s why I come back, always to the recommender systems… So, I think that’s the Achilles heel, on which a lot is based. Then you have a quasi-duopoly in the online advertising space on which online media and newspapers in their digital or hybrid format rely on, because that siphons off of revenue from the newspapers to the online advertising industry, that needs to be addressed.

The fact that you generate data for your newspaper by generating clicks, which means that your headlines and your content have to be based on or leaning towards clickbait and lower quality rather than the type of high quality that a subscriber would rely on, is a problem in that it drives quality down and it’s a problem in that it drives revenue away.

And on top of that, you have the untransparent nature of the online advertising market and there was a report only this week about child abuse material being rewarded by advertising from major brands and from government departments in some cases.

So, these are all interconnected problems and all generated by the attention economy and I think this is the place to look if we want to solve many problems by addressing that one source of problems.

00:48:15 Domen Savič / Državljan D

And who do you see as the leading actors in this field? We have already talked about the civic sector, the decision makers, the companies in the digital rights field… is the situation similar in countering disinformation, or is there somebody who’s leading the pack a bit more compared to the field of digital rights?

00:48:46 Joe McNamee

So, from the digital rights side, there are so different NGOs in the space, have different expertise and different priorities… however, there is a lot of energy in the data protection and freedom of expression fields and there’s a lot of energy in the disinformation field, and I think we all have a shared interest in getting to the root of this, this whole range of problems by requiring transparency, by requiring alternative recommender systems, by requiring an end to the fraud in in the advertising sector, by helping wean newspapers off low quality and back, as some are beginning to do back towards quality driven, subscriber based models.

So, I think there’s a big overlap, I wouldn’t like to pick out anybody as being the champion in in this area, there are a lot of champions, thankfully.

00:50:04 Domen Savič / Državljan D

No, but so my question wasn’t yeah, to sort of put you on the spot… I was just trying to hear your thoughts on… are there differences between the ecosystems, right?

Digital rights, at least in the beginning, but also in in the last few years, had a very strong support from the digital industry sector, that was present in this field to promote their own goals, but to sort of also expand and to sort of normalize the discussion right in the area of disinformation, you kind of don’t have anybody who’s actively or who’s openly for disinformation.

You have various, you know, interest groups that wouldn’t mind if disinformation was flowing freely, right? So, in addressing this issue, is the majority of the burden again on the NGOs, the fact checkers, on journalistic organizations? Or is there somebody else from the field of academia, from the field of state institutions that are helping address this issue?

00:51:31 Joe McNamee

I think I’m not going to be able to provide a simple answer to that because I think anybody that’s working on rigorous implementation of data protection law is working towards an environment where disinformation is less hard to drive through the attention economy.

If they’re in their NGO’s working on implementation of the DSA and for example measures against systemic risks and measures to ensure adequate transparency and transparency of algorithms, they’re not working directly on disinformation, but their success will be a success for disinformation.

There are a lot of people working on analyzing strategies behind disinformation campaigns and these are not… nobody has got the free speech rights to build a billion bots to drown out everybody’s voice that is not a free speech, right?

And there are a lot of people working on analysing the networks and the structures and the technologies behind disinformation that are also working more explicitly in the disinformation field.

I have a particular personal dislike for the word ecosystem as a metaphor, but iIt really is harped in this case, that we’re all in the different battles we’re fighting in the different areas of digital rights, we are all playing our part to make the environment better for everybody.

00:53:43 Domen Savič / Državljan D

OK. And one final question, I know it’s just February, but the annual EU Disinfo lab conference is coming to Ljubljana this year in October, you’re still, filtering through the program proposals, but let me ask you this… why did you choose Ljubljana for the next iteration? Was it because we at some point had a place in the disinformation universe?

00:54:23 Joe McNamee

Well, sadly, every country has got their place in in the disinformation universe at the moment. It wasn’t me that made the choice, so I can’t be precise, but I think from my experience that area of Europe has got an unusually sharp and vibrant civil society as well as experience in a sort of political crossroads.

Ljubljana has got a lot going from a location, it’s a beautiful, small city, it’s got ancient neighboring countries have a strong civil society and everything we heard from local government agencies was also very positive and we’ve had very productive discussions on topics related to us with government officials and those implementing the DSA.

So, I can’t think of any argument that I heard against Ljubljana.

00:55:55 Domen Savič / Državljan D

OK, thank you very much, we’ll wrap it up at this point. Thank you, Joe, for dropping by, for sharing these thoughts, best of luck going forward and if we don’t see each other before, we’ll definitely see each other in in October.

00:56:14 Joe McNamee

Much looking forward to, it will only be my second trip to Ljubljana.

Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!

We sat down with Maximilian Gahntz, Mozilla Foundation’s AI Policy Lead, working on questions around the regulation and governance of AI around the world. Previously, he has also led work around data governance and platform accountability.

Before he joined Mozilla, he was a fellow of the Mercator Fellowship on International Affairs, working on the EU’s AI Act at the European Commission. 

We talked about the currents in the AI policy development, the issues with media representations and political interpretation of the AI issues and more.

00:00:11 Domen Savič  / Citizen D 

It’s the 9th of January 2025, but you’re listening to this episode of Citizen D Podcast on the 15th of January same year. With us today is Maximilian Gantz from the Mozilla Foundation, he’s the AI policy lead, working on questions around the regulation and governance of AI around the world.  

Previously he has also led work around data governance and platform accountability, and before that he was a Fellow of the Mercator Fellowship on International Affairs, working on the E US AI Act at the European Commission. Happy New year, welcome and so good of you to drop by.  

00:00:48 Maximilian Gahntz / Mozilla Foundation 

Happy new year, thanks so much for having me. 

00:00:51 Domen Savič  / Citizen D 

Well, to jump right into the gist of it, we’re obviously going to talk about artificial intelligence, the AI act, the regulatory frameworks and maybe for an opening question is: How will the EU AI Act help or prevent the big tech capture of the AI field that we are facing today? 

00:01:19 Maximilian Gahntz / Mozilla Foundation 

It’s a good question and I think it’s important to also take a step back and think about what the AI act actually is and what it isn’t, because in in the end it’s a Product Safety law, not competition law. So, like the primary intention of the legislators here was never to make sure that there is no concentration of power or to, you know, curb the power of big tech.  

The intention was to essentially make AI products in the usaver and prevent harm first and foremost, so I think that’s just like important context here because obviously there are some implications. There are some tiered rules that might apply to bigger companies, but not to smaller companies, there are some exceptions, for example, when it comes to open source. 

But ultimately, it’s a Product Safety law, so I think if we want to talk about concentration of power and market power here, we need to look at this in a broader context, because there are other levers we can pull and that regulators in the EU, in the UK, in the US are trying to pull as well, and in general, when it comes to the success or not success of the AI Act, I think it’s also important to say that it’s too early to know really because there are so many vague provisions in the AI act is so much implementation work to be done, standardization, secondary legislation, there is a big code of practice process currently underway for developers of large language models. 

So, in the end, what the rules that the AI act put forward will look like, we’ll only know in a few months, a few years, and then how these rules are enforced is actually the long game here, so it’s too soon to judge how well the act is working out at this… we’ll actually have to wait a few years and then also just be… As activist and civil society organizations very vigilant about how these rules are implemented, how oversight authorities are enforcing the law and then what companies are doing in the market.  

00:03:39 Domen Savič  / Citizen D 

So currently like the broad perspective of, let’s say, the AI landscape consists, if you look at it from, from, from a perspective or through the eyes of let’s say a regular digital user, it looks like you have a couple of big companies that are battling over for market dominance and then you have some localized or individual tools that are being used by smaller actors. 

Would you say that assessment of the AI landscape is correct or are there other players who we as activists should pay attention to when we’re looking at this situation? 

00:04:31 Maximilian Gahntz / Mozilla Foundation 

Yeah, I think naturally a lot of the attention right now is on the big AI models and their providers. If you look at what the people in the press are writing about and people are talking about is ChatGPT and Gemini, and what those tools can and cannot do and how they’re implemented into different services, but I think it’s important to keep in mind that, sorry, this is the tip of the iceberg of the AI value chain, which is pretty long and complex, but if we’re talking about the distribution of power and concentration here, it’s important to go down or go up the value chain and think about whose strike dominance in these different markets that are really important for AI.  

Because in the end, obviously like what, end users, consumers, people are going to be using or going to be affected by, it’s the applications and how these tools are put into practice and deployed, whether it’s in like a consumer setting or even by governments, so naturally like a lot of the attention is there. 

But if you think about what you need to build these applications, there’s a lot that goes in there, so you have the big model provider is Google open AI anthropic and they train large language models. If we’re talking about generative AI here, using a ton of data and a ton of computing power and raw processing power and to do that, they need a lot of data. So, who has the benefit here? 

The companies that already do collect a lot of data from other services, for example or that have the capacity to enter into licensing deals with publishers or other rights holders or who have the capacity to build a web crawler who can crawl large parts of the web and download their data and feed it back into the model.  

The other part is computing power and there, if you look at who the big cloud providers are, who provide that computing power, it’s roughly the same companies or many of the same companies that you also see strike being very active in the AI industry. It’s AWS from Amazon, it’s Google… So they actually have a lot of market power in the cloud market as well. There is the chip providers where I think when it comes to a specialized AI chips, NVIDIA controls large parts of the market, I think it’s like somewhere in the 80 to 90% market share range when it comes to AI chips. 

So, the further up you go, you still see market concentration and you can’t really look at any one of those levels in… you do need to look at each one of those in combination, because only that way you will actually see who’s like if. If you want to, you know, put it in a poignant way, like who holds power over AI? 

00:07:48 Domen Savič  / Citizen D 

You previously worked on data governance and platform regulation… How is let’s say the area of AI and maybe more specifically generative AI, how does this differ from the situation that we’ve been having with digital platforms, with social media, with trying to regulate them for the past few years? Listening to you naming all of these actors, it would seem to me that it’s just a little bit of history repeating, right? 

00:08:28 Maximilian Gahntz / Mozilla Foundation 

I think in parts, yes and parts no. Obviously the problem space is still somewhat different, because, for example, if you’re talking about safety and bias in AI, it’s a bit different with many AI products compared to online platforms, social media platforms who have their own problems of bias that’s like deeply enmeshed in content moderation and recommendation algorithms.  

And obviously this is a very salient topic right now, again because two days ago, just announced that they’re going to change their content moderation practices, so I think some of it is similar, but obviously if we’re talking about content moderation, there’s been in the past 10 years a lot of debate around the limits and you know, liberties of free expression and the value of preserving free expression that’s been politicized in different ways. 

It’s probably something that will come up in the AI debate as well at some point, and we’ve already had some discussion around like, do chat bots display political biases, for example, but that’s not the core of the issue. 

So I think there’s definitely lessons to be learned and we should look at the different experiences we’ve made in different digital policy and other policy fields in the past ten, 20-30 years to learn more like new debates on AI, but it’s not going to be, you know the same thing all over again, because there’s just different equities, different changing political circumstances, obviously. 

What the European Commission and the new US government and the new UK Government are going to be talking about in the coming years is going to be somewhat different from our political focus areas have been determined in the years.  

So, there’s a… I’d say there’s a shift in the grandeur political context as well here. 

00:10:38 Domen Savič  / Citizen D 

Yeah. And is that going to, let’s say, positively or… I know I’m a bit naive… positively or negatively influence the way we’re addressing all of these… you’ve already mentioned some of them, you know, dangers or pitfalls of regulating this field? 

00:11:00 Maximilian Gahntz / Mozilla Foundation 

I would say yes. Umm, if you listen to policymakers in Brussels right now, for example, there is a lot of emphasis on European competitiveness. If you’re talking to people in the UK, there is a lot of focus on enhancing productivity, if you’re speaking to people in the US, there is a lot of focus on national security and AI, and especially in relation to China.  

So, I do think that changed significantly… and if you’re an advocate, you kind of need to adapt to that, because fundamental rights and consumer protection to the people who are emphasizing that often aren’t the primary concerns. The challenge for us here is to figure out, how we can still insert those concerns and make sure that like despite a broader competitiveness agenda, which is, you know, OK to have, that despite that fundamental rights aren’t kicked to the curb, but rather it’s like integrated as a fundamental part of that. And then the same is true in other countries and other contexts as well. 

But I do think it will like drastically change how we’re having these policy debates in the coming years because I think… in the past European Commission mandate, for example, curbing platform power and discretionary… that a lot of platform companies, for example, were able to make that was very important to decision makers.  

I think it still is as we’re currently seeing playing out debates around, you know, Elon Musk and Mark Zuckerberg and how they’re inserting their voice in the EU but overall, the debates are going to be different, and there’s going to be a lot more talk about competitiveness, national security productivity, etcetera. 

00:13:14 Domen Savič  / Citizen D 

OK. And how would you, let’s say rate the impact of AI on these topics; on national security, on the competitiveness, on the labor force. Because on one side you have these public conversations where the AI group is saying, “OK, AI is here, it will solve all of our problems, increase productivity…”, on the other hand, you have skeptics who are saying, “Oh no, the current state of AI market isn’t sending very strong signals that these tools can actually do the jobs, they’re being highlighted for” and then you have the pessimists try who are saying, “OK, they’re going to take our jobs and we’ll be left with nothing”.  

Based on these three camps, where would you see yourself in?  

00:14:24 Maximilian Gahntz / Mozilla Foundation 

It’s good question, probably it oscillates a little bit as well. Historically, I’ve probably been like a bit closer to the skeptics’ corner, but I think it’s also important to disentangle like a couple of different things here, right? 

There is a lot of hype around AI, is it like a bubble? Maybe yes, but I think we’re also seeing that at some things it might be… if you’re talking to people in the creative industries, you’ll hear that you know, like AI isn’t going to replace everything, but it might replace somethings. If you’re talking to someone who’s writing… I just listened to a podcast with an essay of a composer who writes jingles for advertising sounds like, you know, generic background music. 

That’s something that AI can like do reasonably well by now and I think there are more and more areas like that where we like starting to figure out what it might be good at. There are many areas where it’s not good at and it’s like it’s made out as this magic bullet and eventually, we’ll find that maybe it’s not that magic bullet. 

So, I think we’ll also over time, just have to figure out what the nuances here are and also stop using AI as a blank canvas that we can like project anything we want onto. If you look at like party manifestos in Germany, we’re voting in a couple of weeks and parties just released like draft manifestos, everyone’s mentioning AI in very vague terms, it’s supposed to solve a lot of challenges. 

Will it solve those challenges? Probably not, at least not the way that people think it will. 

So, I think overall, as the debate is maturing and the hype is maybe dying down, we’ll be able to have a slightly more grounded conversation about where AI is going to really change things. And obviously also I mean with that… the benefits and the risks of AI, because I think, it’s not necessarily a net positive in every respect of AI is really good at a job if we’re talking about applications, used for surveillance or used at the European borders in the migration control context. 

There are things were like AI might actually be quite effective, and we still might not necessarily want it deployed at scale in those contexts, which I think is also important. 

00:17:14 Domen Savič  / Citizen D 

How do we direct the public debate or you’ve mentioned political, political campaigning before the elections, right… So how do we direct the public debate around pros and cons of implementing AI in in in particular fields as opposed to just going all out and saying “OK, just put an AI on it’ll work”? 

Who should contribute to debate? Or how do you see different actors meshing together on this topic, like the public framing of the power or lacks of artificial intelligence solutions? 

00:18:04 Maximilian Gahntz / Mozilla Foundation 

I mean, big caveat here is that I’m not a campaigner. I’m a policy wonk, and I’m constantly struggling with, like the messaging aspect of doing this work. How do you actually change like a dominant frame and a policy debate? And I would say often it’s not up to… It’s just something that happens and that suddenly drastically changes how debates are being had in Brussels or DC or London or Berlin or wherever. 

Case in point is like working on AI policy and working for example on the AI Act was a very different job before the release of ChatGPT and after the release because that that really kicked off the frenzy, right? 

That really changed the discourse and changed the emphasis of a lot of policy debates, although I think it’s important to educate decision makers about how this technology works, because there is a lot of talk of AI as the sort of like mystic thing, and it’s computing. It’s non-deterministic, which is why the unpredictable stuff happens, but in the end it’s hardware plus software plus data. 

So I think demystifying AI is important and also, you know being mindful of the hype merchants are, if you want to call it that, who the players are, who have a vested commercial interest and playing up this technology and talking about future scenarios such as “AI is going to upend everything,” because they might stand to benefit from it, and because it’s good marketing. 

And then also, make sure that there is space for people to think and talk and debate about it’s like where we want AI and where we don’t want it and then how… because obviously this is going to be, I do think it’s going to be relatively pervasive in different areas of life and sometimes it might just replace human decision making. 

And then it’s a really important question in what domains? What decision? What’s at stake there for people? Especially given that there is often like a tendency to experiment with those technologies on marginalized, vulnerable, disenfranchised communities and to make sure that even when the zeitgeist currently dictates that the EU needs to watch out for being competitive and having a strong position vis-à-vis other geopolitical actors that doesn’t that that doesn’t void the need to talk about risks and fundamental rights because ultimately, that’s the baseline for a flourishing society, right? That people retain their rights and liberties and that technology does not interfere with that because it’s not a deterministic development that we’re having here. 

And ultimately, like it’s, it’s up to us to help shape how this technology is being developed, how it’s commercialized, how it’s deployed, that’s not something for like a few techies to decide, something for a broader debate within society in the long run. 

00:21:51 Domen Savič  / Citizen D 

You’ve mentioned policy development and we’ve been talking about the EU, but what are your thoughts if you, let’s say, compare the US, the EU and China in this regard in terms of regulatory frameworks or policy developments that are focusing on the artificial intelligence. What are some in your opinion main differences and what are some similarities in that regard? Is there a common ground that literally the world agrees on? 

00:22:35 Maximilian Gahntz / Mozilla Foundation 

I’ll have to say I’m not an expert on… there are people who know much, much more about tech policy and tech in general in China, so take what I’m saying with a grain of salt.  

I think over the past maybe 5-6 years like one very common narrative we’ve heard is that of the three different pathways and tech and tech policy, right? It’s the US, that’s completely laissez faire deregulatory. We’ll just let our companies do whatever they want and it’s China, which is putting its foot down and being very prescriptive and then Europe is the middle ground where we’ll have innovation, but also preserve European values.  

There’s probably still some merit to that, but I also think it’s a bit of oversimplifying things because if you look at the US at this point, at least for the past couple years, it hasn’t been completely lassie affair. 

Obviously, there’s always been a lot of gridlock in Congress, so Congress doesn’t really pass a lot of new laws also on AI, even though they introduced a lot of bills on AI in the past year or two, but there are many different states which are trying to regulate AI so if we are talking about EU, that’d be like a lot of Members state regulation in the absence of European wide or in the US Federal Regulation. 

If you’re talking about California and I think Colorado, there is a lot of debate and some adopted laws on AI already, there was a big executive order by the Biden administration in late 2023 that put a lot of rules on, for example, government use of AI… So, I would say the idea that like nothing is happening in the US is slightly outdated.  

Obviously, less happened in the US than in EU, for example and then we’re going to see what’s happening with the new administration, the Biden Executive order on AI is probably going to be repealed, but it’s kind of hard to foresee right now what the Trump administration will do on AI because they’re also competing camps. 

Obviously there’s a lot of venture capitalists who supported his campaign, there is also Elon Musk, who’s like a big AI safety proponent and talks a lot about existential risks from AI, so it’s going to be interesting to see that play out, but overall, we can probably expect a bigger shift of focus away from like AI ethics and civil rights to national security and American competitiveness. 

And then in China, when people in EU have always been saying like, you know “We’re the first to regulate AI”, that’s also only semi-true because China has put forward rules and regulations for platform recommender systems for example, but also for large language models a while ago and I can’t really, I can’t really speak to the exact nature of those rules and how effective they’ve been, but China’s legislators and policy makers are very active on a AIaround the world, not just in EU.  

And that’s just talking about the US, the EU and China, right, where that hasn’t even mentioned like a lot of what’s going on at the international multilateral level because the G7 has adopted its code of conduct on AI, the OECD, the UN is increasingly inserting itself into policy debates and they’re building out that capacity. So much more will happen at the international or transnational level as well in the coming years. 

00:26:50 Domen Savič  / Citizen D 

And speaking of this happening on many levels, is there like a way to sort of dissect the actual chamber of power, where decisions are being implemented and are put together? 

So again, listening to public debates, listening to political speeches, listening to discussions from the industry, it seems that they all, these different sectors, all have their perception of where the real decision is being made, right?  

The industry is saying, “Oh no, you know, the regulators will just come up and mop up everything that we’ve been doing for years”, the regulators are saying “No, no, no, these decisions on regulatory frameworks are being written by us and we are the smart ones” … so how would you see this flow from innovation towards regulation that is actually working out for the general population? 

00:27:59 Maximilian Gahntz / Mozilla Foundation 

I think, at least for us in Europe, it’s also just democracy at work. And with all its flaws and benefits and intransparencies, because obviously if we learn about our political systems and high school or at university, studied political science back in the day, you get a very formalized process that you can draw upon a sheet of paper, and that’s how it still works.  

But obviously, like the reality is much messier and there’s a lot of lobbying which I would also use as a value neutral term because there is a lot of people lobbying for important and good things as well. There is a lot of backroom, there is a lot of public back and forth depending on how politicized an issue is.  

And I don’t think that necessarily varies with AI from other policy… Obviously, that’s the one I’ve been following the most, but I think the old blame game about how regulators are over regulating, U.S. companies are profit driven and are thus trying to preserve their interest and then wherever civil society and other actors fit in this, I think that’s largely the same. 

Obviously if we’re looking at the AI Act for example or like other tech policy files in EU and elsewhere, I do think it tends to be an uphill battle for civil society and everyone who works in civil society and has tried to engage on tech policy files will know this because there are other actors with much more resources to put into this and big public policy and government relations teams and agencies who just got a leg up.  

So, the challenge of coming together as civil society and despite maybe having fewer resources, making your voice heard. That’s just one that won’t change, we just have to come up with strategies and scheme together. 

00:30:18 Domen Savič  / Citizen D 

Speaking of this exact issue, working as an activist or as an advocate, do you think it’s better to focus your intentions or your energy on the local level or the national parliaments and governments or is this a Brussels game exclusively? 

00:30:43 Maximilian Gahntz / Mozilla Foundation 

Definitely not a Brussels game exclusively, I think it really depends on the issue and on political context. Obviously, if you live somewhere where your local politician or the rapporteur for a big legislative proposal in the European Parliament that gives you different leverage than like someone from a very different country in EU being responsible for that file. 

I think there’s always value in engaging locally, but I think, when it comes to tech policy, I will say a lot happens in Brussels and Brussels has its own logic and structures and I say that as someone who’s doing this job from Berlin. 

There are people who are much more in the center and I’m more of a tourist who every once in a while, goes to Brussels. I think it really varies and it really depends, but it just like requires you to be become familiar a little bit with how the process works and like who the people are that you want to talk to affect change. 

00:32:03 Domen Savič  / Citizen D 

And looking forward and slowly wrapping up our conversation… What are some of the things you are paying attention to as the year progresses, right? We’re doing this in in the beginning of January, so what are some of the policy developments or interesting topics that you are focusing on in the field of artificial intelligence? 

00:32:31 Maximilian Gahntz / Mozilla Foundation 

I mean, I think there’s the legislative side of things because we’re still waiting to see what the new European Commission will do. For AI, there’s debate about an AI liability directive, which basically sets rules for a civil liability for AI developers and we’ll see where that is going or whether it’s going anywhere.  

The UK is working on an AI bill and I think they will release on Monday an “AI opportunities plan”, where is currently unclear what exactly that means, but it’s more about the investments side and less about the regulation side.  

We’ll have a lot of debate about copyright rules, for example and how those might need to change or not and what the other plumbing is in our like copyright and rights enforcement system to make sure that there’s sort of balancing of interests between tech companies and creatives and rights holders and journalists, for example. So, I think that’s the nitty gritty policy side.  

Obviously, there is an overarching development that we’ll all be watching, which is like how is the new US administration going to work? How are they engaging with for example EU in general, but also on issues like tech because this has the potential to be very contentious and we’re already seeing strike some posturing right now around, for example, the EU imposing rules on US tech companies and the Trump administration may not like that, and there’s a lot of US tech executives who are also like starting to play into that narrative a bit more. 

So, we’ll actually have to see, you know, whether UK policy is going to suddenly be part of trade policy debates because if the you start sanctioning American companies, that might or may not trigger a response from the US… So, I think like at the geopolitical level, things are going to get much more complicated and probably a little messy and I think it’s important to get in that a better understanding of how that might pan out and then what different actors, both at the government level, also at the civil society level can do here. 

00:35:07 Domen Savič  / Citizen D 

And before we wrap, up just one more question… so AI went through several periods in the history going back to the 70s in in the 20th century. Looking at the current situation, the players, the regulatory frameworks, the situation in the in the industry, can you sort of describe how the next AI winter looks like. 

00:35:42 Maximilian Gahntz / Mozilla Foundation 

I you I think you’d have to ask someone else. I think that whoever gives you an answer to that, it’s a creative exercise. There’s… it’s hard to predict how hype might die down or not, like what technical developments are looking like… It’s good to remind yourself of what, like the market forces are here and like, think about some of the trends in AI, because I think like, what’s safe to assume is that this debate and this landscape is going to continue to change. 

I think there’s a lot of people who talk a lot about acceleration, there’s a lot, some people who talk about a bubble bursting, but I think if you look a bit closer and like you know what might happen within those things. 

There are so many different scenarios of how this might pan out, AI might just become a commodity that gets like at some point much more efficient than it is right now, much less energy intensive. We won’t be talking about big models as much anymore because we’ll all have a small model on our phone and then it’s just, you know, regular computing. It’s a new technique and it’ll be good for something, then bad for other things. 

I think that’s one scenario, obviously, like we might also continue on “bigger is better” trajectory that we’ve been on for like the past couple years where companies are just continuing to throw a lot of money and capital into this to build ever bigger and bigger and more powerful models. And then we really have to also talk about resource constraints both like economic resources but also natural resources because we know that developing AI models and actually using them as super energy intensive, has real environmental impacts and it’s no coincidence that a lot of big American technology companies are currently talking about or already doing it, bringing nuclear power plants back on the grid, building their own power plants. 

So, I think… everything might get bigger, everything might get smaller, or somewhere in between. And then what? I think well, to be honest, we’ll have to just like be wait, have to wait and watch and be vigilant because I think as someone who’s like not at the on the technological front line, but rather like an observer of the technical ecosystem, to then start thinking about like what societal and political consequences of that are you just got to make sure that you understand what’s going on in order to be able to act and then ideally shape where things are going. 

00:39:05 Domen Savič  / Citizen D 

And that goes for, I would assume that, the self-aware AI. So, I would love to hear a few more thoughts on this matter, because this is constantly popping up every time the industry it seems needs a PR boost, Altman and others are starting to talk about self-awareness. 

So, is this a feasible or is this a realistic path or is this just something that the PR department needs to put out just to ramp up the groupies? 

00:39:46 Maximilian Gahntz / Mozilla Foundation 

I mean I have at this personally, a lot of skepticism, but ultimately, I don’t know, right? I think on average, humanity is probably not super great at exactly predicting the future, so I’d have my doubts that, you know, Altman, or some tech executive or, you know, some civil society advocate is actually going to predict what’s going to happen in 2040.  

I do think there’s a lot of hype, I do think there is a lot narratives that are being pushed because someone has an interest for pushing them. I think what’s also clear is that, like the technical community, like very much doesn’t agree on what’s happening there or maybe some voices that get a bit more airtime because they’re like, you know, powerful CEOs or because they want a Turing award and are famous computer scientists. And I think these are all positions that have merit, we also need to look at them as a prediction of what might happen rather than someone actually knowing what’s going to happen. 

I think there’s been a bit too much of that in the press, I think we all just got to learn how to deal with uncertainty, to be honest, because… It’s just an AI, it’s a bit crazy right now and everyone’s imagination is running wild. In the end, we… just got to see which person, just like by pure coincidence, happens to be right. But that only applies if we’re talking about the very long term… if we’re talking about what’s going to happen in two years, that might be different. But I’m not going to make a prediction about 2050 ever. 

00:41:35 Domen Savič  / Citizen D 

…unless it the year is 2048, right? 

00:41:39 Maximilian Gahntz / Mozilla Foundation 

Yes, we can meet again in 2048. 

00:41:44 Domen Savič  / Citizen D 

I’ll pick you up on that. Thank you, Max, for dropping by, this was the first 2025 episode of Citizen D, we publish an episode every month and we focus on different range of topics from digital policies, regulation and human rights advocacy. See you next month! 

Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!

Za konec leta postavljamo piko na i teme mladih in informacijske družbe s Katjo Koren Ošljak, ustanoviteljico zavoda Vsak in asistentko na katedri za medijske in komunikacijske študije na Fakulteti za družbene vede, ki se že skoraj deset let na različne načine ukvarja s fenomenom mladih in informacijske družbe.

V pogovoru se dotakneva fenomena neukih digitalnih domorodcev, razlik med digitalno in medijsko pismenostjo ter vpliva tehnologij na spreminjanje medijskih navad mladih kot posledica pandemije, vseprisotnih družabnih omrežij in drugih digitalnih fenomenov.

Dotakneva se še generativne umetne inteligence, razlik med vzgajanjem digitalnih potrošnikov in digitalnih državljanov in razvojem politik informacijske družbe.

Podcast Državljan D je podcast za produktivno preživljanje časa, v katerem igramo vlogo državljana. Državljan D v pogovorih s strokovnjaki določenega področja informira in aktivira. Da se. Gremo naprej!

We sat down with an award-winning investigative journalist Mackenzie Funk, whose book, “The Hank Show: How a House-Painting, Drug-Running DEA Informant Built the Machine That Rules Our Lives” tells a story about the origin of surveillance capitalism we know and hate today.

We talk about the book and the man behind the story, but we also touch upon his legacy, the surveillance capitalism that stems from data economy and big data intermediaries and the way we have to address this on a local but also on a systemic level.

Also included in the conversation are the critiques of predictive policing, the issue of almost unchecked business of data analytics and the way forward.

00:00:06 Domen Savič / Citizen D

Welcome everybody. It’s the 23rd of October 2023, but you’re listening to this episode of Citizen D podcast on the 15th of November same year.

With us today is Mackenzie Funk, an award-winning investigative journalist whose second book, and it’s a mouthful, “The Hank Show how a house painting drug running DEA informant built the machine that rules our lives”, tells a story about, well, I’m just going to say it, it basically describes the birth of surveillance capitalism.

Hello, Mack. Welcome to the show. Would you say the description is accurate?

00:00:44 Mackenzie Funk

Yes, you’re right. That is a mouthful. I never had an easy way to explain what the book was about except that it was… Yes, it was the person who we’ve all never heard of, who started so much of this world we now live in.

00:01:01 Domen Savič / Citizen D

And before we start with the book, so the first question I’d like to ask is before this… You were so you’re an award-winning investigative journalist and you focused mostly, or you focused mostly on environmental stories, global warming and stuff like that. What made you pick the the Hank Asher story?

00:01:21 Mackenzie Funk

I think, as crazy as it sounds, that surveillance and climate change are very similar in some ways, especially to an audience that doesn’t know very much about either one.

They can seem very dense and very boring at first and so you need to find a way to tell a story that is not just about the facts of it. So, in the case of climate change, it was just about here’s the science. And so, for many years before, I was working on my climate change.

You would have these arguments in the United States between the people who believed in the science and the people who did not, and it was not a very good way to win the argument, just to be right, just to just to have the scientists have the right facts.

And so, I was trying to find a way with the climate change book to tell a story that would maybe show the stakes show why people should care.

When it came to surveillance, that was my same technique as I focused on a person because I figured even if you’re not interested in privacy or surveillance, you might be interested in this person. But the second thing is that they’re both these big systems and the reason they that some people find them hard to… they find them boring or hard to understand is because they’re it’s so complex.

But there are these big systems that seem to fall the heaviest on the people who are poorest among us and the most vulnerable. It’s becoming more obvious with both climate change and privacy that the poorest people in the world, the poorest people in each of our countries, those are the people who are bearing the brunt of this and that there can even be winners in these new economies from climate change and from surveillance capitalism.

And those winners are not the poorest, they’re the usual winners in our societies, and I find both of these both climate change and our lack of privacy to be accelerating some of the worst inequities in society.

00:03:25 Domen Savič / Citizen D

Before we jump into that, I saved this part the last part of the show. But first let’s talk about the book.  So how did you find this guy? Hank Asher, the guy who… I’ll let you tell the story or a brief recap, but this was…

Reading your book this was the first time I heard about it and I am working in the field of, you know, digital privacy and digital activism for the last fifteen to twenty years

00:03:56 Mackenzie Funk

Yeah, I would not say that I was by any means a privacy expert when I started this, but I certainly paid attention from reporting abroad, especially in in places like China and Russia, I was very careful about… I tried to understand where my information is going and who will see it and even after the Snowden revelations in the United States, I became a little careful about even what my own government was looking at.

And I had never heard of Hank Asher. His name first came up when my… it’s a complex story, but a magazine approached me. An editor I knew, and he said I have a story for you.

And the story was about this group of people who were trying to stop child predators as they called them, and they were using this software built by a person named Hank Asher, who they described on their website as the father of data fusion. So, I was looking at this group, and I saw that note on their website. And I said, who, what?

And that is that is what it began my journey, and I then googled the name and then I saw that he had been a cocaine smuggler, that he made a fortune multiple times and then lost it in Florida in the 80s, I saw that he had this crazy story that he was involved in the security build up after 9/11 in the United States.

That his technology undergirded some of the biggest surveillance companies in the world and certainly in this country, and that it was just this character nobody had ever heard of. And I found that amazing.

But what really did it for me was when I stumbled upon his obituary page because he had died by the time I heard his name and the things that people wrote about this man.

Just on his online obituary website that his company set up for him and then another one that the Funeral Home put up are just not the kinds of things you see written about most dead people you know.

You read obituaries, you read the comments people make, and they say nice things about people. But the ones that they said about Asher was like, he changed my life. He paid for my kids’ college. He fixed my broken teeth, or he yelled at me. He swore at me. He was the craziest person I ever met. He changed… He changed this country forever. He changed my life forever, and he just seemed to have this outsized impact not only on privacy, but also on the people around him.

00:06:39 Domen Savič / Citizen D

The book is full of these anecdotes or happenings in in his life when you, I mean, the title sort of tells a pretty good story. So, you have a guy who’s been drug running for, for the DEA, or who was an informant for the DEA, but at the same time, he was saving lives helping people, you know chase down, child molesters and stuff.

But what was in your view the thing that surprised you the most? Like when you were doing, when you were doing the research for the book?

00:07:16 Mackenzie Funk

Of course, everything about I got perhaps a little too obsessed with understanding what exactly he was doing in the 1980s when in the United States, the center of all drug smuggling was Florida because of its proximity to Latin America, to Colombia, to Jamaica, to the Caribbean, and I became obsessed with that. But that was interesting, but not surprising.

The biggest surprise was how these open records laws in Florida were a good thing in this country. We wanted transparency and the States, especially after some scandals, Watergate and others. They decided to make sure that the public could know what the government was doing in their name and so many states opened up their records, opened up their books so that citizens could see what the government was doing and people like Hank Asher, in states like Florida, which had very open public records, exploited this.

They were able, using this state law, they were able to go in and get all of the driver’s license records and all of the vehicle registration records, marriage records, birth records, divorce records, all the housing record. Everything, everything you can imagine that the government, local government would use to sort of have a citizen move through their life.

This was a public record and because the governments wanted this to be they, they wanted citizens to know what they were doing. They opened it up to a new species of person, which is these data, data aggregators.

And the technology, by the time Asher came along in the late 80s and 1990s, you were suddenly able to scoop all this up and make sense of it in a way that I don’t think you could have when these laws were written.

So that was the big surprise that this, this very progressive policy was became something very different and would… say if we compare the Hank Asher’s period where he was buying up or gathering all of these data points and putting them in a in a database in a searchable database, would you say there are similarities if we draw a comparison with Mark Zuckerberg or Elon Musk or any of the other big intermediaries, digital intermediaries’’ bosses…

00:09:54 Mackenzie Funk

Yeah, I have similarities and an important difference in the big trick with Hank Asher’s era. We’re talking in the 1990s and early 2000s is that they are going out to all these different databases and trying to make sense of these little tidbits of information they get from each one, they’re trying to connect all of them to a single person.

And to do that, they ended up assigning a tracking number to each American citizen and each American resident, and also many people across the world, so that each of us has our own our own bar code or something like this.

And then they had very tricky rules that became algorithms or and now are different through machine learning, they’ve perfected these but different rules that assign a new data point that comes in to an individual with some degree of faith that this is going to the right person so I am not the only Mackenzie Funk in the United States.

It turns out there are many of us and they tend to know when I’ve moved or when I’ve bought something. And they do so because they’re able to see, of course, my address, but also the people I’ve lived with and my age, my gender, these kinds of details. So, they can with some faith put this new data point with me.

Someone with a name like John Smith, it’s two very common names in this country, it’s much more difficult, and yet they would have ways to make sense of this.

Contrast that with what Zuckerberg built, which is you have to use your real name on Facebook, and the moment you you do, you become your own aggregator. Zuckerberg did not need a fancy algorithm or a fancy set of rules to attach each new data point to an individual because we did that for him, we did it by logging in under our real names.

We did it by putting Facebook on our computers and on our phones, and so it almost turns the problem on its head from the perspective of the of the aggregator and I think that’s the biggest difference with what’s come out today and Asher’s day is that back then the problem was information was coming in from all sides and we need to find a way to really attach it to a person.

In Zuckerberg’s era, it was how do we make people build their own databases and then give us access to it and I think right now in a maybe post Facebook era, we’re a little bit in a more of a hybrid space between the two. The same problem Asher solved is now being solved again from the from the data broker perspective, they’re trying to do it again.

00:12:51 Domen Savič / Citizen D

And just to continue your train of thought…  So, what are some of the lessons that we can learn from his story right regarding data economy regarding these data points regarding data aggregation regarding I don’t know use of data in, in policing and surveillance?

00:13:19 Mackenzie Funk

Yeah, I. Well, one thing is that Asher, I think had high minded goals with what he was doing and why he was doing it. Yes, he wanted to get rich, but like many people in Silicon Valley, say 15 years ago, he kind of believed he was the good guy and that he was aggregating all this information to do good things, and that included, especially in his case, using this information to go after child predators, to go after child molesters and to help the police do that.

Then after the attacks of 9/11, he used it to go after supposed terrorists, he thought it would help the police solve crimes. He thought it would eventually, you know, help companies determine lower rates for people who had good driving records, for instance, or otherwise look good. And it was hard for Hank Asher to see the downside of what he built.

But I think he was an early stand in for people like Zuckerberg, who may have claimed high minded principles. At the beginning, you know bringing the world more connected or whatever the slogan was in in the early days of Facebook and yet all this stuff has such a dark side that if you build this this weapon, eventually you’re not going to be the one in control of where it’s aimed or it will become take on its own.

And with Asher, you could see the dark side of this pretty quickly, his technology was used not by him, but by the people who took control of his first company to change the results of one of the most important elections in this country by kicking black Americans off the voter rolls in Florida, they did this in the name of keeping fraud out of elections. But the end result was that, it appears that George Bush beat out Al Gore in the 2000 election in Florida and therefore won the entire country.

You can draw a line from what Asher built to that moment, not something that he would have wanted. He actually was in favor of the Democrats in favor of Al Gore, but that’s how his technology was used, I don’t think that he would like how it was used much later in Florida, again disenfranchising and taking away the vote of black Americans.

I don’t know that he would have liked how it misidentified various Muslims in this country as somehow being terrorists. But it did. I don’t think he would like how people were wrongly accused because a police officer who was badly trained in this information would pick the name of somebody from his database and say “Oh, this has got to be our suspect”, but that’s happened.

So, I think that’s the parable you can draw from what he built is that this stuff has such power and eventually its creator won’t be in control of where this power goes.

00:16:44 Domen Savič / Citizen D

Do you think… what you just said… that this lends itself to a theory that, you know, technology is neutral and that it matters who’s running the show if I can borrow a phrase from the title of the book, or would you say that technology has some inherent biases and inherent problems built in, and the driver behind the steering wheel isn’t the only, let’s say good or bad switch?

00:17:20 Mackenzie Funk

That’s a good question. I think the reality is that it is both things, I think the driver matters quite a bit, but also the technology can have something built in that makes it inherently worse or better, and I don’t think that it has to be one or the other.

And that again is the lesson I would see here, if you think about what Hank Asher built and why he built it, in some sense, it’s a suspicion machine. Keep in mind that this is a former cocaine smuggler who suddenly is wanting to work with the police, he knows that he has something hidden in his own background, he thinks like a criminal.

He knows how the how the Drug Enforcement Agency, the DEA, has gone after other drug smugglers, and he’s seen how they put information together and draw connections and so what he kind of built a machine that would, that would show hidden connections between people or between people and what they were trying to hide. For instance, say their assets.

And a lot of the machine was built based on his own psychology and because of that, it’s something that surfaces, things that most Americans would think would be hidden, like if you’ve lived with the same person three times in a row and they know that because his machines have your entire address history and they can see “Oh, these people were roommates.”

What if in a time that this country was a much more anti-gay, what if that was your husband? If you’re a man and that’s your husband, but you were keeping your relationship a secret… Well, that’s the kind of hidden information that his machines would surface.

And they basically remember every little thing that you might want to hide, or think would be hidden. And that’s based on in part who the maker was and how he started to think about what would be useful to investigators and what criminals or other people might be hiding… So yeah, there there’s some inherent danger to the technology, but also how it gets used is very much depends on who’s in control of it.

00:19:44 Domen Savič / Citizen D

And if I can just backtrack a little bit, you’ve mentioned that after Watergate, the government wanted to be transparent and open. So nowadays you have digital activists, you have human rights activists arguing for more transparency in the government works in police works in in these supposed closed system systems.

But on the other side, transparency on its own… I sometimes get the feeling it it’s not doing much good, I would say, in a way that OK, sure, you open up all the databases and people are drowning in these data points. And every once in a while, you have this Hank Asher type of type of person who, you know, makes sense and a little bit of money on the side from all of these.

So would you say, how would you argue or what would be the correct way to sort of argue for transparency, but at the same time not argue for, OK, you just open up the floodgates and let it all out. And, you know, we’ll figure it out as we go…

00:21:08 Mackenzie Funk

I am no expert in the data privacy laws in the United States versus Europe, but I do know that you have a much better understanding that data collected should be used for the purposes it was collected and not for other things, and one of the big things that went wrong in the United States and I think continues to go wrong and round around the world is that this idea of transparency also means that companies can purchase or otherwise acquire data it was never meant to be used for, say, advertising.

When it’s something that a local government has collected, but because there are no controls on how it is used. Two or three purchasers down the line, then it becomes transformed into something very different.

So I think the answer to much of the transparency and privacy problem, which is real, is some sort of control on… is this used for what the person who gave up their information to their local government, for instance, intended if I give my information to the Department of Motor vehicles in my state here, I don’t expect that they are going to then sell that information on to LexisNexis, the data broker.

I don’t expect that LexisNexis will then send that information onto immigration authorities and that the immigration authorities then could come and come to my home if I am not a say, I’m not a citizen and I don’t have the right papers to be here. Those are not things I expect when I go to get a driver’s license. And I think some sort of… something like that is built into the laws, can go a long way to helping… I tend to agree that transparency isn’t enough, I tend to think that that governments are collecting too much information and they are keeping it far longer than they need to because then.

If there is transparency, if there are transparency laws and it happens to be in the database still, well then, they might have to give it to whoever asks for it. If they don’t collect it in the first place, or if they don’t retain it for a long time, that’s often safer.

But I think the tension is always going to be there, but I think a lot of it is about the very the gut check. Is this what I thought was going to happen to my information?

00:23:47 Domen Savič / Citizen D

And do you see things changing like now, like in the past? Like maybe after the pandemic or even during the pandemic, it seems that you know the data industry, all of these big digital intermediaries, they finally lost its, I should say user-friendly halo, right?

And people are sort of pushing… in the United States and also in Europe and also in other places across the globe, they’re pushing for a for a different approach to these to these issues of privacy of, you know, companies knowing almost everything or everything about the person.

So do you see that happening as we move away from from, let’s say, the COVID pandemic and go forward?

00:24:36 Mackenzie Funk

Yes, I do see it happening and I think again, I know the situation in the United States better than I know in Europe, but in the United States, the big change was after the election, in the first election of Donald Trump.

After the 2016 election and Facebook’s role in that election, the Cambridge Analytica scandal, those things seem to very much change the public perception of these of social media in particular.

And then you see more controls on our phones, both Android and Apple have done a better job as to not letting other companies track us all over the web all the time. Not that they’re perfect, but we have a lot more control than we did eight years ago.

And I think that is that is a good thing. That said, I think we’re coming back, especially in in the United States from 20 years of almost no control on privacy and, and so even if there’s some incremental gains on the digital space…

A lot of this information is already out there and a lot of the information that companies like Hank Asher built and those are big parts of LexisNexis and Thomson Reuters and TransUnion, those databases are still there and so they’ve got20 going on 25 years of information about each of us and they never got rid of it. They didn’t purge it, they kept it. And so even if governments are collecting less, even if our phones are giving a little bit less information about us, the length of that history that some of these data brokers have on us and that still includes Facebook, that really matters because they can get a sense of where your life is going and where you’ve been and those are… That’s pretty important information when they’re trying to help other companies make decisions about your life.

I looked at healthcare in the United States, doctors began to look at not just your clinical conditions, but at the conditions of your life now in general, where do you live? Do you have access to a vehicle to get to your doctor’s appointments? Or a bus or a train?

Do you drive a certain kind of car? Because certain kinds of cars can be, you know, can you can judge someone based on their car. You can kind of guess what their general health is. Where do you get your information? Because if you’re getting all your news from Facebook versus of newspaper, you might be lower information and you might not follow the directions of your healthcare provider as much.

That kind of information or even things like where did you live when you were growing up nd was that area worse in terms of air pollution… that kind of information, going toward healthcare decisions now is very… You’re not going to easily get away from that just because your phone isn’t telling Apple or Google exactly where you are right now.

00:27:59 Domen Savič / Citizen D

And just to follow up… So do you think that all of these data points about the person or the information packets you just described, do you think these are actually used by people or by companies by industries collecting it or do you feel that this data economy is basically I’m not going to say a mirage, but basically something that looks nice, looks fancy, looks I don’t know, important, but at the same time you just get these, you know, data dumps that that nobody uses and then, you know, when somebody hacks a big, you know pharmaceutical or health related database, then it becomes a problem, right?

00:28:47 Mackenzie Funk

Yeah, I think they are using it. In fact, I know they are using it, but most of the Hank Asher’s products, most of the world’s biggest banks and most of the biggest companies use those products. Most of the law enforcement agencies in this country use these products and I’ve seen… I mean, imagine it’s not just the little bits of data, right? It’s tied to a person’s identity, all the information you might want to know about them.

For figuring out how much to charge them for insurance or like I said, healthcare decisions, or if you’re a police officer trying to track someone down, where do they live? Where have they lived? Who are their friends and who are their relatives? If you’re trying to find them, where would they go?

That kind of information is in there if you are an immigration officer trying to track people down as, as could happen in in this country and has happened in this country. It’s the same kinds of questions, and also when did they arrive? When does their name first appear in this database? Does this show that their recent arrival in this country?

And I found evidence that it’s being used all the time. I don’t know how banks work in Europe, but when I log in from a new location, say I have a new computer or a new phone and I want to access my bank information sometimes, it will send me a bunch of questions: which of these street names have you lived on or which of these streets have you lived on in your past? Who of these people have you lived with? Which of these people have you lived with? Little details about my biography that only I would know.

You would see things that stretch back 15-20 years into my history and that’s a Hank Asher product built right in and used for something that I think is pretty benign. It’s making sure that you are really you, but it’s a show of the reach that these products have everywhere in the background of our lives in this country. And nobody really has any idea…

00:31:10 Domen Savič / Citizen D

So, the case that you just described with the banks it perfectly I think describes the issue between privacy and security, right? So, you have these data products that are a boon for security, but at the same time they are literally almost or not even almost literally killing our privacy. Right?

So, do you see this conflict? You know, resolving in in the in the future, in, in any shape or form, or do you think we’re just going to go back and forth between, you know, this is too much security. We want more privacy or this is too much privacy, we want more security?

00:31:54 Mackenzie Funk

Yeah, I don’t see it resolving and nor do I see the sister of that one, which is the privacy and convenience question. And we see of course that that one all the more with the new AI products that say Microsoft or Apple is coming out with how much of your information do you want to give to these companies, especially if they’re uploading it to the cloud to make your life more convenient to complete your emails or calculate the distance between when you have to leave so you can get to your next meeting or to reschedule a meeting.

These APIs, these AI products can already do this if you let them have access to your calendar or all of your emails and I find that scary and very interesting because I think a lot of people will choose convenience over privacy here and that the privacy controls will always come late. And as for security and privacy, it goes back to your earlier question about are they actually using these products? Do they actually work?

I think we’re finding that as much as, at least in this country, as much as they were used for, say, counterterrorism, or for predictive policing or all these things where the idea was, if you get enough information about everyone in in a country or in a city, and if you really build these algorithms to help us predict who’s who among the general population. You can stop crime before it occurs or stop a terrorist attack before it occurs.

So far, that hasn’t worked out as well as it was supposed to so far… yes, police officers are using Hank Asher’s products to say, figure out someone’s address history or who their relatives are, but they’re not using it in sort of an AI powered way… let’s let’s just give everybody a score, and I mean they are using it, but when they’ve used it, it hasn’t worked very well.

So, a lot of them have dropped it. So, a lot of the predictive policing projects have been dropped in major U.S. cities because it didn’t work as well as they thought it would. It just turned out to be racism at scale.

It just turned out to wrongly tar huge swaths of a city with the idea that they might have a higher risk of being a criminal, which didn’t always match up with reality and was certainly unfair to those people who were caught up in the algorithm.

So, I do think that there’s been… Yeah, I sometimes think that is showing that, especially with law enforcement and counterterrorism, that it’s not as useful as they thought.

00:34:49 Domen Savič / Citizen D

I was asking this because, you know, we in so this podcast is a part of the NGO with the same name and we focus on privacy, security and amongst other things, surveillance capitalism. And we’ve been doing this investigation in the CCTV cams that are set up by municipalities across the across the country and we realized that it’s basically… it’s a promise of security.

They never deliver because the whole pipeline that follows detection, we have somewhere around 500 CCTV cameras across Ljubljana, and at the same time, they’re not doing much in terms of crime prevention, but they’re doing a lot in terms of, you know, privacy invasion, right. And if you see, if you look at across the world, you see that, you know, some technologies work in particular cases, but they sure don’t work in in other cases, right.

So the debate around security and privacy becomes very targeted, I should say. Or you know, you have to look at individual cases, individual technology and so forth. So, would you say the data, the data policing or the policing with the use of data, is that something similar or would you say the success rate is much higher or higher than that?

00:36:18 Mackenzie Funk

Hmm… we have this similar debate with traffic cameras and surveillance cameras here. I would say that the use of data in policing, it has been useful and hat Asher brought and what his companies bring to local police departments has accelerated the kinds of investigations they would do.

For instance, they I was talking to these detectives who worked in Florida in the 1990s or in the 1980s, about how they would go about their casework and they would go and track down each of these individual pieces of information from different police departments across the state, and they would not necessarily have access to say the stuff you could get from credit cards, which always have a good address history, or they would or they could get it, but they would have to get a court order and the court order would take weeks. They could get someone’s phone records to know who they’ve been calling. But that again would take weeks or months and so to just to get enough information to try to track down a suspect in a crime, it would take them a very long time.

And all of them found these products that Hank Asher built to be transformed and to help them do the work they were doing much faster and I as much as I was skeptical of where the technology has gone.

I wanted to be fair and recognized that that it really did help some of these police and do their jobs and I and I think in a way that it didn’t necessarily hurt the privacy so much, no more than they already could. It was just happening faster, but things changed.

When Asher’s company and others began to make this more algorithmic and to start assigning scores to someone’s criminality or their likelihood of being a terrorist, and I think where the a lot of the failure has been, is that when taking humans out of the loop or in saying that we need all of this so that we can solve these crimes, I don’t think you need all of this. And you certainly don’t need the scoring technology to help you surface people you think might be more likely to be criminals than others.

It’s very different to try to predict crime than it is to try to track down suspects in a crime that already occurred. And I think that once they started to try to predict the future using this data, that’s where many of the biggest harms have happened and then when it goes away from these narrow uses. I can see that someone investigating a murder, for instance, might it might be more fair to give them this information than someone investigating a minor crime.

One thing we have in the United States, not just the surveillance cameras around the around cities, but are the license plate readers on all of the police cars and on many city cars all over the country. And many of those were installed by a private company and that private company has taken all these scanned licenses and they’re able to give predictions is about where someone will be.

For instance, if you always see someone’s car parked in front of a certain address at a certain time of day, you can predict that they’ll be there pretty soon, and some of that, that just seems like too much. Like who actually needs that?

You know, who needs that is real estate companies or banks trying to use data for things very different than policing and these same companies were selling to the police and they were also selling to business.

And I think the selling to business part doesn’t need to happen and the retention of all these scans over the course of many years, does not need to happen.

Other people using it were immigration authorities trying to track down people. In this country, who might who might not have come legally, and my opinion on that is that in the United States, there are so many people who came to the country without the right papers and there are only so many they’re going to want to arrest.

And it seems like if you are going to try to prioritize people, it’s the worst idea to go after people who have been here for so long that they might have children who are citizens in this country and who have real lives and who are living like any citizen, if you have to prioritize.

And yet these systems, they will surface the people who are living the most, who are not trying to hide right? The ones who engage in banking, the ones who properly registered their car and got a driver’s license and got insurance, bought a home.

All that information, that all goes into these systems. And so, it turns out that the people who are not trying to hide, who are living the non-criminal, very average lives are the ones who are easiest to find in these systems. And so, if they’re used for these purposes like that for immigration enforcement, for instance, that just seems like a total perversion of what what they’re for.

00:42:08 Domen Savič / Citizen D

I have two more questions before we wrap up. So, the first one is basically a continuation of this debate.

So why do you think these technologies, these data silos that are basically telling us where a person is going to be at a particular moment, why do you think the usage exploded in literally every corner of our society, so if you like, take Hank Asher, for example, right at in the beginning, he was basically helping out the police to catch pretty basic criminals, right? And then, the usage extended to anti-terrorism and then it extended to anti child molester.

So why do you think the people are so excited about copy-pasting the same basic solution to, you know, places or fields that that may not have very much or much in common with. You know, the previous or the original field that this technology was used in?

00:43:20 Mackenzie Funk

That’s a good question. I don’t know the full answer, but I can say that, before this was used for policing, the original product that Hank Asher had and that many of these things grew out of was not policing, but insurance. Private insurance companies trying to get a sense of who am I selling this insurance policy to and how can I reduce my risk?

And the idea of reducing risk is basically look at the past to try to understand the future. Try to do a calculation about what someone has done before in order to predict what they’ll do in the future, and then if you can do that and whoever does that best they’ll make the most money, if you’re an insurance company to those clients who are riskier and they’ll charge much less money to clients who are not really going to have any problems.

So basically, they’re getting money for nothing, and that that’s the whole idea. It seems like with Hank Asher, they took this idea of risk management from insurance and applied it across all of society.

This idea that you can just look at someone’s past to predict the future, or more broadly look at people like this person and what they’ve done in the past to predict the future. That it’s a very seductive idea, because imagine for one if you can predict the future, you can make money off it or you can protect against whatever coming danger is.

So, we all love to know what’s coming down around the corner. Human literature is full, is full of stories about time travel, or the person who could see the future or the seer and whoever knows what’s coming is almost the most powerful person in the world. And that’s what these systems promise to police and everyone else and counterterrorism investigators is a chance to predict the future.

And maybe they get it right, for the most part, but the problem is, this isn’t advertising. In a lot of these systems are not used just to send somebody an advertisement for a tennis shoe, or a handbag they might want. These are used to make life and death decisions about that really affect people’s lives, who gets insurance, who gets a job and who gets arrested and getting it right most of the time, I don’t think should be good enough.

I think it’s very different if you send the wrong ad to the to somebody, no big deal, you’ve wasted a little bit of money. But if you arrest somebody because you think they’re going to do something wrong, that’s a very big deal. And so, we can’t have the logic of insurance or the logic of advertising which is similar to that of insurance, which is your risk pool just… If you get it 70%, that’s great, if you get 50%, that’s great.

We can’t have that logic be the same logic for the rest of our lives.

00:46:36 Domen Savič / Citizen D

And since you mentioned healthcare… OK, two more questions. Since you mentioned healthcare, why do you think this data surveillance or data economy isn’t regulated as well as health field, right?

So, you’ve just mentioned 70% isn’t good enough… imagine if you would have those options going to a doctor, right. And the doctor would go well, you know, I can heal you, but I can also kill you, so let’s see, let’s see what happens next.

Why do you think the like, the regulatory frameworks, and this is the second part of this question, still heavily rely on the end user. So you’ve mentioned Apple and Android building all of these protections in in in our devices in our, you know, online services and stuff, why do you think there wasn’t a bigger, let’s say, systemic push for, I’m not going to say corporate protections of our privacy, but let’s say state protections of our privacy.

So why are we at the end of the day David versus the Goliath?

00:47:54 Mackenzie Funk

Good questions. The first one… healthcare. It’s very easy to understand that these are life and death decisions. It’s very easy to understand that what happens with the doctor or at the hospital is going to change everything for a person.

I think it was harder for people to understand and imagine that what happened with privacy would have such impact, and this goes back to your earlier question about why did you go from or how do you go from climate change to privacy.

Early on and climate change, everyone was talking about polar bears, right? They couldn’t imagine they knew they cared, but they couldn’t imagine that really that climate change would really affect their lives.

They couldn’t imagine the wildfire smoke or the glaciers disappearing or anything really changing for them. So sure, they cared about it, but not in a way that was visceral and close to them, not like a healthcare decision.

And I think with privacy, a lot of people are still stuck in the imagination that it’s these advertisements that are that it’s about advertising that it’s about these ads following you around as you surf the Internet and that, yes, it’s a little bit creepy, but what does it really have to do with my life?

How does it really change my life and that the public has been slow to wake up to the reality that it has everything to do with your life and more and more the information about you is out there being used to make decisions about your life and what opportunities you’ll be given.

And once that understanding is there and it’s becoming more there than maybe data will be regulated a little bit more like healthcare because we’ll see that it’s the same level of decision.

As to the second part of the question – why is it always the end user? Why is it not systemic? I think it’s related a little bit, I think it especially in the United States, people think well, it’s your choice to use Facebook or it’s your choice to give up this information online and if you really care, you can do something about it and it’s almost the companies that still dominate the privacy landscape, which are American companies are built with this American individualist ethos.

That we are a country where we pretend that the individual has lots of power and that we are supposedly self-reliant and that the government shouldn’t step in and regulate, and that if you really care, you’ll take action on your own. And of course, that’s nonsense. That’s nonsense with climate change and it’s nonsense with privacy. The idea that people would have any way to meaningfully counteract the most powerful forces in our lives without banding together without asking government to do what government is supposed to do, which is step in when it’s a collective problem, it’s nonsense, but it’s still permeating.

I think the debate and that’s the biggest issue is that that that we’re stuck with this individualist sort of American ethos.

And another point on that, the idea that there is any meaningful choice anymore that you that you can somehow avoid the digital economy that you don’t have to give up your information if you don’t want to, that you don’t have to use these tools that their tech companies built, that’s not… I mean for, for young people, for anybody who wants to participate in society as it is now, you have to be using these tools and therefore you have to be giving up information for the most part.

And that’s not really a meaningful choice, when a when a gun is at your head.

00:51:54 Domen Savič / Citizen D

We’re trying, we’re not always successful in wrapping up on a positive note, so please if you have like a like a… I’m not going to say an uplifting message for the end and I’m not even going to ask you about how’s the current US election going and what do you think will happen after November in this field… but is there a place to or is there a thing that you notice regarding data economy regarding you know these protections, that is, let’s say improving in the last couple of years? So, is there a light at the end of the tunnel?

00:52:34 Mackenzie Funk

Yes, I do think as we’ve talked about that the public is much more skeptical of Silicon Valley and of these technologies in the United States in particular, then the public was five years or eight years ago.

And I think it we are slowly waking up to the fact that this has life and death consequences for people that privacy is not just an ad that follows you around the Internet and that knowledge, even if it’s too late, even if it’s a little slow, will eventually translate to change, I think.

I think it must, we’re not stuck just doing whatever these companies have us do entirely because we are, they’re still voting in all of ours countries and there are still elections, and those elections, if voters care about these issues and voters care more and more about these issues, I think there can be change and little things.

I saw from Hank Asher’s technology for instance, the utility companies, we’re talking Internet companies and electricity companies, water companies for many years they were all getting together and they created a database of all these different customers and what their address was, what their names were, and they did it in the name of making sure they were credit worthy so you couldn’t switch from one electricity provider to another.

And if you were someone who didn’t pay your bills that that they would know about that. But they started taking this information and selling it to the data brokers which you would never imagine that you move into a house and you decide to get electricity and because of that some corporation knows where you are or that they sell that to police.

And that was one of those things that once it was uncovered in the United States, a few years ago, there was outrage and they stopped selling it to the data brokers that went away. It’s little things like that, there is a change.

There are in this country and I know also in the EU there are lots of small fights against overreach by the surveillance companies, not all of them are victorious, but there are more and more, and I think that is a sign for some hope.

00:55:10 Domen Savič / Citizen D

Excellent, that’s a perfect way to end this debate about the really horrifying situation everywhere. The book is called The Hank Show, thank you Mackenzie for dropping by, for, for sharing your thoughts, it’s really nice to see these ideas of privacy and the ineffectiveness of the security capitalism echoing around the world.

Thank you so much, dear listener, this has been the Citizen D podcast, we publish an episode every month, so feel free to subscribe and we’ll talk to each other next month. Thanks again, Mackenzie.

Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!

With us today is Meredith Whittaker, president of the Signal Foundation who serves on its board of directors. She was formerly the Minderoo Research Professor at New York University (NYU), and the co-founder and faculty director of the AI Now Institute.

She also served as a senior advisor on AI to Chair Lina Khan at the Federal Trade Commission. Whittaker was employed at Google for 13 years, where she founded Google’s Open Research group and co-founded the M-Lab. In 2018, she was a core organizer of the Google Walkouts and resigned from the company in July 2019. She now runs Signal, the leading global privacy-orientated NGO.

00:00:55 Meredith Whittaker / Signal

But I wouldn’t actually say that the walkout was the very beginning for me, the walkout was a culmination of a lot of work, a lot of thinking, a lot of conversations that I’d had over more than a decade. And the walkout also wasn’t just me. It was thousands and thousands of people. It was apparently the biggest labor action that has happened in tech, with 20,000 people leaving work in protest, you know, against the unethical business conduct at Google and against a culture that persistently valued some people more than others and developed products that often caused serious risk for those who were devalued, so to speak, due to that culture and those design decisions.

I think the walkout was one way in which throughout my career, in many, many ways I have endeavored to be accountable to my analysis, I have endeavored to do what I can to change things when I saw them going in in a bad direction, but I had worked for many years and many different ways, from the inside trying to influence trying to shape policy and many of these things I still do… So again, I think the walkout wasn’t the beginning. It was one manifestation of a theory of change that looked to collective action from below to remedy some of the dangers and harms of the concentrated tech business model.

00:03:52 Domen Savič / Citizen D

00:04:42 Meredith Whittaker / Signal

Conservatives, liberals, leftists… recognized simply structural check on toxic capitalism and labor power involved, the workers having some say in what they work on and how. I don’t know that this is individual so much as going back to some of the basics and recognizing that we have an industry that is making some decisions and putting revenue and growth above the common good in ways that could be really, really dangerous given the power and information possessed by this industry.

00:06:11 Domen Savič / Citizen D

00:07:15 Meredith Whittaker / Signal

We are asking people to validate our claims and we are making insofar as possible everything available for them to do that and I think that is why Signal is so trusted, because in fact we, we are going above and beyond to be trustworthy in a way that most actors in the ecosystem can’t or are unwilling to for a number of reasons.

00:08:36 Domen Savič / Citizen D

00:08:55 Meredith Whittaker / Signal

And so, it is difficult to do the opposite. We actually end up having to rewrite parts of the stack, so to speak, in order to enable privacy, in order to reject data collection as a norm. So that is difficult because we are swimming upstream against a massive current in a trillion-dollar industry, where privacy has not been something that was prioritized and trust around privacy is certainly not been part of the business model. Now it’s also difficult or related to that it’s difficult because there isn’t a business model for privacy at this point in the tech industry, and this is one of the huge harms that we are, we are grappling with.

The profit motive is oppositional to privacy, data collection is oppositional to privacy. So it’s difficult from that perspective in that we have to really think about our structure and protect ourselves from the imperatives of profit and growth, not necessarily because they’re bad in and of themselves, but because following those imperatives, would at this point lead us down a path towards surveillance toward data collection.

So, this is why Signal is structured as a non-profit. This is why we really go out of our way to take the incentives for surveillance off the table when it comes to Signal again. So, we’re structured for success in the long term, so we stay laser focused on our mission.

00:11:29 Domen Savič / Citizen D

So, is it hard for you, for Signal to argue for privacy when faced with a fake dilemma of choice between privacy and security?

00:12:17 Meredith Whittaker / Signal

And the motive is that there are some among governments and law enforcement who feel that the fundamental human right to private communication should not be available to people online, that there should be no communications network that is not tappable, that law enforcement or governments aren’t able to surveil.

And I think that is… It’s just simply incredibly dangerous, and it flies in the face of the long-standing expert consensus that knows there is no way to create a backdoor, create a way in that only “the good guys” can access, that anytime you create a flaw in these infrastructures, anyone with the tools and expertise to exploit that flaw will, and so you are corroding the very same cyber security measures, the very same private communications networks that your government also relies on, that your law enforcement also relies on and you are making those vulnerable to hackers, to hostile nations and to whoever else might want to infiltrate those.

So, it is a very pernicious line of argument, but I don’t think it’s always in good faith and I don’t think that we’re ever going to win this battle simply by being correct simply by force of argument. We’ve been correct for multiple, multiple decades. The facts have not changed, but the will to create some magical formula that lets the government spy on everything does not seem to die.

00:14:35 Domen Savič / Citizen D

00:15:09 Meredith Whittaker / Signal

And I think that that provoked a kind of counter reaction. You saw a number of the platform companies, you had iOS and Android adding encryption to their operating systems, you had a turn to privacy from the industry that wanted to in effect, save their reputation if we’re going to be cynical about it, one to distance themselves from government spying by adding privacy features, and immediately after that in 2015, you see a showdown between the FBI and Apple in the US over the encryption on the iPhone.

And you begin to see an escalating campaign, as it were, to undermine the privacy guarantees that have been put in place post Snowden, most profound among these is Signal and the Signal protocol.

Often people who are perhaps a bit parochial or confused that want to undermine and walk back these changes. Now there are many kind of dynamics that I think have helped or hindered this, but I see this as one more salvo in an ongoing battle and no sign that we are losing the war… and in fact in the last couple of years we, those of us in the privacy world who are pushing for these fundamental human rights, have had a number of wins, have pushed back on a number of pieces of very bad legislation in the face of often incredibly emotional and compelling narratives that are difficult to fight against, particularly when someone is bringing a heartfelt story to the table and then you’re on the other side debating about the nuances of cryptographic mathematics or something, right?

00:18:50 Domen Savič / Citizen D

00:19:37 Meredith Whittaker / Signal

There are great services for private e-mail like Proton, but Proton is also interoperable with other major e-mail services like Gmail or Outlook which means that if I e-mail somebody on Gmail from my Proton account, Google has all of that information shared on their servers, right?

Similarly, if I were to pay for privacy, I’m paying the subscription fee so that you don’t collect my data, say, but I e-mail somebody or I connect with somebody, communicate with them in in some way and they don’t pay for that, I would also not be kept private, so you know communication networks are really something that shows us just how interdependent we are when it comes to privacy.

00:21:25 Domen Savič / Citizen D

00:22:19 Meredith Whittaker / Signal

We need to ask to efficiently fulfill the role of governments in these core services is, what type of technical infrastructure that is not controlled by monopoly actors? Would we need to be able to govern technology in a way that is more democratic? What type of governance structures do we need to put in place that allow participation at the level of design and features and how a technology behaves, how it respects or fails to respect fundamental rights.

And those are questions that are ultimately very exciting and I do think we are in a time when there’s no longer any debate over whether this business model is good or bad. You even have institutions like Combinator, which has done, you know, arguably as much as anyone to cement and promote the toxic Silicon Valley business model, is now coming out and saying like, “Hey, we’re actually not very into big tech. We’re looking at little tech now. We want to promote the, you know, the small players!” and whether that’s good faith or not, I think that really shows us that there is a sea change in terms of sentiment and that there is an opportunity to think through.

How would we dismantle and disarm the centralized power that is held by these platform companies? You know, what would independent cloud infrastructure look like? What would independent communication networks look like, what would interoperable protocols that enable more flexibility and independence at the application layer look like and how do we find the capital to fund these things and maintain them forever over time?

And how do we put in place governance structures that don’t behave like the boardrooms of big tech with their focus on profit, revenue and growth over everything else, but have more civic minded duties and processes that are working to aerate the tech ecosystem and make it more amenable to building technologies that actually serve beneficial futures.

00:25:46 Domen Savič / Citizen D

00:26:33 Meredith Whittaker / Signal

So, I think in a sense that when I say we need to get down to a more local level, that’s not fetishizing the small and the local. That’s really recognizing the function of these platforms, the role they play vis-à-vis government services, vis-à-vis commerce and communications really does vary across contexts and that the people in those contexts are almost certainly best positioned to answer some of these questions.

We don’t want to repeat the mistakes of one size fits all billion-user platforms, but simply do that kind of interventions right, because there is no one-size-fits-all.

00:28:09 Domen Savič / Citizen D

00:29:11 Meredith Whittaker / Signal

In the US, which I know is different than many other places, but it’s also generally funded through philanthropy, so you don’t have long term sustainable funding in most cases you are at the whim of whatever foundation or your donor might think it is important at that moment, and of course that is also susceptible to trends and to whim and to hype, which makes it very difficult to pursue a long term strategy, particularly when you are rowing upstream against vested interests that may frankly, may have a lot more access to some of the leaders in philanthropy than some of the activists who are on the ground doing real work but not being seen and appreciated.

So again, I think the political economy of NGO work and civil society needs a lot more scrutiny. And I think we need to be a bit bolder in frankly demanding the kind of support and capital that we need to do this work. And there’s a lot of really good ideas out there, really good architectures, incredibly brilliant thinking around how we could build tech differently, how we could build more respectful tech, but an idea is not the solution.

An idea is a possible template and what is not generally understood or let’s say respected is just how much work and how much money it costs to build reliable tech. It’s never just built once. This isn’t two guys in a garage, who come up with something genius and the world changes. No, it’s two guys in a garage, a really good idea and then billions of dollars of capital and hundreds of thousands of hours of labor who make that idea real, to maintain that idea in a volatile and dynamic environment and to do that forever or until that idea dies, or that tech doesn’t exist.

I think we also need to reframe our understanding of tech and recognize that we can’t have Sam Altman be the only one who’s talking big money, right? We also need to recognize that we’re serious about this change. We need to be at the table, and we need to be demanding a cut of that.

00:32:19 Domen Savič / Citizen D

00:32:57 Meredith Whittaker / Signal

We can look at things like race, science, that couched structural inequality as neutral biological destiny, they were just observing differences and then determining what those differences meant in ways that were pernicious and harmful for the world. I think we need to question narratives of neutrality to begin with, and then particularly in tech, I think that this has been a conflation of computational technology with scientific progress, which has been promoted by the tech industry.

The reason we’re all suddenly using Google or we’re all hosting on Amazon is not because those companies were successful during the primitive accumulation stage of tech but is simply because what they discovered is that significant scientific advance and they are introducing that to the world and as such, they bear no responsibility for that as such, that what they are doing is neutral and inevitable, it cannot be changed and as such, if you were to question it or if you were to say desire to regulate it in a way that wasn’t beneficial for those companies, you are anti-progress or anti-science, you’re putting your finger on the scales of human advancement, and I think that narrative has done as much as anything to really chill our ability to grapple with and meaningfully regulate these technologies over the past number of decades.

00:35:49 Domen Savič / Citizen D

00:36:50 Meredith Whittaker / Signal

You know, these are people who imagine themselves as always, in a position of power, and thus don’t generally question in the type of levers of power that we are creating and the way those could be misused, if somebody with more pernicious intentions were occupying their seat, so, I think this is an age-old pathology and it’s why we need to hold anyone who has a position of power to incredibly stringent standards and recognize that it’s really not personal, but that if you are going to take that kind of responsibility, you need to be held accountable and the people who are worthy of that responsibility should be embracing that.

00:38:05 Domen Savič / Citizen D

00:38:10 Meredith Whittaker / Signal

00:38:34 Domen Savič / Citizen D

00:39:18 Meredith Whittaker / Signal

So, there are many, many things, I would say 99% of the things that we would like to do are not things that we choose to do, because we do really value focus and we balance, we think long and hard about new features, we think about whether we can build those features in a way that that meets our very strict privacy bar, we think about whether those features are useful to people.

For example, Signal introduced Stories a couple of years ago, similar to Instagram, but our Stories are actually private, right? And while they’re not a hugely common feature in the US, they’re massive in South Asia and in Brazil and we were hearing from people that pick up Signal and it feels broken, because it doesn’t have this feature that has been a core way of communicating among the people who use Signal there.

So, it’s a lot of conversations, a lot of collaboration with our Chief Product Officer Clancy Childs, who is very brilliant and very experienced and has been working on messaging now for over a decade, who really has a lot of instincts there and we try to do some market research.

We do don’t collect user data, we don’t collect telemetry and analytics the way almost every other communication service does, so we often don’t have or we almost never have, the kind of signals that our competition does, but we do have other ways of fetching information and doing user research in the field that gives us a sense of how are people using Signal, what might they enjoy and we go from there.

00:42:08 Domen Savič / Citizen D

Now the end result for a normal user is gif search, right? It looks exactly the same as if we just shoved it in there and said “Hey, we’re giving all the data to Meta”, but in fact we spent orders of magnitude more time, creativity and rigor doing that than the competition.

00:44:14 Domen Savič / Citizen D

00:45:04 Meredith Whittaker / Signal

00:46:02 Domen Savič / Citizen D

00:47:20 Meredith Whittaker / Signal

How do we put the disinformation to rest and how do we push back against it? Well, we really do go above and beyond. We are open source. You can validate our claims if you have the skills, you can go into our repos, there are people who watch every commit we do and analyze it on Reddit, looking for new features, discussing what we’re doing, submitting bug reports, so we are one of the most scrutinized apps and one of the most scrutinized and audited cryptographic protocols in the world.

How does that translate to a popular message? Well, we are out there talking about Signal security as much as we can. We spend a lot of time when these campaigns of disinformation that are saying that Signal is a CIA asset or whatever the nonsense is. We spend a lot of time pushing back, even though we know this is fully fabricated and kind of, I would say almost lazy. We also recognize the stakes are really high, that there are people who don’t have the expertise to validate these claims themselves and can get really worried by those claims.

So, we also ask that the overzealous community of people who may be amplifying those claims or making them in service of getting attention or going viral. please find something else to do, because digital security is life and death for a number of people who use Signal in authoritarian context, and these kinds of rumors can have real harmful impacts on people, even if they are completely baseless.

00:49:54 Domen Savič / Citizen D

00:50:32 Meredith Whittaker / Signal

Podcast Citizen D gives you a reason for being a productive citizen. Citizen D features talks by experts in different fields focusing on the pressing topics in the field of information society and media. We can do it. Full steam ahead!