The fact that the NSA has access to so much data about you is scary and bad. The fact that they denied collecting said data is even worse. Still, we should not be surprised.
IT technology makes it easy to transfer, store and compute information. Thanks to IT, we are able to communicate faster, cheaper, safer and, in general, with more people than ever before in history of communication. Over the years, a global, massive grid of interconnected networks became a reality, and as its complexity grew, it became evident that it would needed monitoring for technical glitches, tuning and security.
Tools for deep packet inspection, logging and metadata collection became vital for IT and IT security teams to do their job, i.e. keeping a certain level of control over their perimeter and networks.
Web and email filtering software also became indispensable to organizations worldwide, as they needed to scan the traffic to and from their organization for unsafe content like porn, hacking tools, keywords that might indicate unauthorized exfiltration of confidential data, and so forth.
Organizations can justify the use of such tools with the need to ensure that no company secrets or intellectual capital leaves the company's "premises", and with the wish to avoid non-work related content to enter their network. Both are valid reasons, and these tools come in handy.
As you members of IT teams well know, being able to access and browse through all that collected data introduces great temptation. And most of you - if not all - succumbed to it at one point or another, and took a peek at some of it: logs detailing Mr. B's visits to porn sites, the online underwear shopping that Ms. D does during lunch breaks, and more.
Of course you knew it was wrong, but you did anyway, right?
What makes it different, then, when a government implements similar tools to protect their country's interests and assets, and just like you, succumbs to the temptation of taking a peek (or, in this case, a long look)? Can we blame them?
Most certainly not. Itīs their country, and they can do whatever they wish to if it's within the limits set by UN and international law.
When the US government (and UK's, and most likely those of several other NATO member states) decided to monitor the use of Internet within their borders (including of course data in transfer), they also decided to keep it a secret.
Unlike citizens of China, Iran and Syria, who know about national surveillance efforts and expect their Internet activities being monitored, US citizens (and to some extent also the Europeans) did not actually, truly believe this could happen in their own country. Security professionals used to joke about it, but many of them were also unpleasantly surprised by the revelations.
And therein lies the problem.
In Europe and the US, we think of ourselves as the forefathers and the protectors of democracy - run by the people and for the people. We elect people we trust, and we trust them to make decisions on our behalf. We may not always agree, but we trust them to know better and to do better.
We put our lives and livelihood in their hands, effectively trusting our leaders with our lives. More importantly, we trust our leaders to tell us the truth.
Well, at least we did until now.
Instead of discussing the need for online surveillance and monitoring, the US government - initially, and for a long time later - denied engaging in such activities. The same happened in the UK, and other European countries, some of which are still in this denial phase.