UPDATE 2: On Monday, December 30th, Der Spiegel magazine published additional information about the techniques allegedly used by NSA TAO to infiltrate the technologies of numerous IT companies. As a result of this new information coming to light, the Cisco Product Security Incident Response Team (PSIRT) has opened an investigation. Customers can stay informed of the progress of this investigation via the previously posted Cisco Security Response.
December 29th – An article was published in Der Spiegel today about the alleged capabilities of the United States National Security Agency (NSA) Tailored Access Operations (TAO) organization. The article says that TAO “exploits the technical weaknesses” of Information Technology products from numerous companies, and mentions Cisco.
We are deeply concerned with anything that may impact the integrity of our products or our customers’ networks and continue to seek additional information.
We are committed to avoiding security issues in our products, and handling issues professionally when they arise. Our Trustworthy Systems initiatives, Cisco Secure Development Lifecycle, Cisco Common Crypto models, and Product Security Incident Response Team (PSIRT) and Vulnerability Disclosure policies are all industry-leading examples of our commitment to our customers. This is central to how we earn and maintain trust.
At this time, we do not know of any new product vulnerabilities, and will continue to pursue all avenues to determine if we need to address any new issues. If we learn of a security weakness in any of our products, we will immediately address it.
As we have stated prior, and communicated to Der Spiegel, we do not work with any government to weaken our products for exploitation, nor to implement any so-called security ‘back doors’ in our products.
UPDATE 1: Customers seeking additional information may refer to the Cisco Security Response.
Additional Resources:
Cisco Trustworthy Systems: http://www.cisco.com/web/solutions/trends/trustworthy_systems/index.html
Cisco Secure Development Lifecycle: http://www.cisco.com/web/about/security/cspo/csdl/index.html
Cisco Security Advisories, Responses and Notices:http://www.cisco.com/en/US/products/products_security_advisories_listing.html
Cisco Security Vulnerability Policy:http://www.cisco.com/web/about/security/psirt/security_vulnerability_policy.html
Cisco Blogs on Security and Cryptography http://blogs.cisco.com/tag/crypto/
These type of unfounded allegations may become an ongoing challenge for all major U.S. networking technology companies in 2014.
Infonetics Research recently reported that cloud service providers are already investing in additional security measures — as a direct result of customer concerns raised by the reports about NSA activities. Therefore, the economic backlash from American government spying could have a significant negative impact on growth.
Unfounded: Der Spiegel is a nice newspaper and I read it each morning. NSA spy my country, Swift (banks) and Belgacom (a telephone company). Through Belgacom they spied the UE parliament.
“Therefore, the economic backlash from American government spying could have a significant negative impact on growth.”
And only when this hits enough companies will anything change.
‘Unfounded allegations’: Is it safe to assume that you haven’t read the article in question? Here it is, and you don’t need german to see that it does have substance:
http://www.spiegel.de/netzwelt/netzpolitik/interaktive-grafik-hier-sitzen-die-spaeh-werkzeuge-der-nsa-a-941030.html
Tip: Click on the red buttons in the image.
I wouldn’t say these are unfounded. do you live under a rock of work for the feds?
Given the current US laws and practices, it is not advisable for non-US companies to use IT hardware, IT services and data hosting from US companies. Even if Cisco does not like it, they will be forced to support US government programs for spying, mass surveillance and cyber attacks.
Yeah, but Google, Yahoo, Facebook, etc., did this whole playing-dumb, “never heard of it” thing with the PRISM revelations as well (Larry Page’s blog-post denial is still up and running). Then we later heard that these companies got paid tens of millions of dollars in PRISM compliance costs, have DITU interception equipment on company premises, and have facilitated a sophisticated surveillance protocol with real-time multimedia capabilities.
So forgive us if Cisco’s own round of denials rings hollow. We have no idea whether you’re innocent, lying on behalf of the government, or lying to cover your own behinds. What we do know is this: You’re a billion-dollar company that could raise a real stink if you wanted to on behalf of your customers. Let’s see what you end up doing. My guess: A bunch of nothing.
You’re already tanking in China. Now it seems you may experience a sudden drop in shippable orders as well. I hope your shareholders are paying attention.
Utterly disgusted and never buying Cisco products again. I’ll be throwing my own Cisco products in the trash when I get home from work, and I’ll definitely be raising this issue at my next company meeting.
SpinDetector,
Ouch.
We fight to earn your, and all our customers’ trust, every day. If you are uncomfortable accepting our words, I’d ask instead that we be judged by our actions. We have made our security policy, including what we do and don’t do, very public. We invest millions in building security into our products, and billions on security products themselves, starting when a product is conceived and going all the way through its development, manufacture, and deployment. We conduct extensive testing for security vulns, managing and disclosing them professionally when they are discovered.
We know we’re not perfect, even with all of our hard efforts in design and build, so when an issue is raised, we provide updates for the benefit of our customers and respond directly to questions – including questions such as the ones popping up now.
While this may not be enough to convince you, I hope it gives you an appreciation that we work every day to make sure attacks of all types, and attackers of all kinds, are not successful and if we find out otherwise, we take it very, very seriously and respond just like we’re doing now – transparently.
–jns
John,
Nobody doubts that Cisco does its best to defend its customers against civilian adversaries. But when it comes to a state-sponsored adversary, the picture changes dramatically.
We have no idea whether the NSA has “legally” compelled you, against your wishes, to do its bidding. We have no idea whether this response of yours has been compelled under threat of imprisonment. This is not paranoia — the other billion-dollar companies have provided precedent of gross deceit. They said one thing, and we later found out another, after they’d been given enough rope to hang themselves with.
If you’re innocent, I feel really bad for you, but you yourself must realize that it’s hard to know whom to trust in this ongoing saga. If Cisco is innocent, this is an opportunity for Cisco to be THE company that uses its financial clout to stand up and say “NO” on behalf of all the citizens of the world who lack a voice.
If not out of altruism, do it out of enlightened self-interest: Cisco execs must realize that their children, grandchildren, and great-grandchildren are going to inherit a world that’s ripe for government overreaches, if not outright tyranny.
ToJohn,
I completely appreciate how hard it is know who to trust, and trusting “us” takes a lot of work that we have to do, it’s fragile, and when something like this happens it really does break trust even if we (ala Cisco) didn’t do anything. I know that, and I’d be asking us hard questions to which is why I’m doing my best to answer all of them on behalf of all us here.
A few key points, and I’m just a plain speaker from upstate New York so forgive me. Cisco is under no secret US government orders as of the time that I write this (nor were we at any time beforehand to the best of my knowledge and that covers 11yrs here), nor has the US government (or any other government for that matter) compelled us to secretely do something to our products to weaken them in any way.
Why do I say that? Obviously one of the reasons I want to say it is because it’s true, but the second reason is, as an senior executive at Cisco, it better darn well be true if I say it. If it weren’t true and I said something like that, I’d get myself in hot water in a lot of ways – especially with all of you. Ask anyone who knows me, and I think you’ll find I take my integrity very, very seriously. And, I am one of the people that should be in the know if we were under such orders 🙂
I just thought I should state it as is, plainly, so you can read it at least once as can others.
peace,
–jns
Thank you so much Cisco, due to your passive response to all kinds of crazy allegations, we decided to trust you folks 100000000000000000000%, so we will stop buying anything from your competitor and focus exclusively on Cisco products.
We are 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000% our data is safe from NSA, from any foreign governments.
Can’t wait to see the stock pop into $40 after Feb 13th.
Hi John,
Apart from the technical measures you’re talking about, as an industry leader Cisco is positioned to influence the government and its agencies. But while AOL, Apple, Facebook, Google, LinkedIn, Microsoft, Twitter and Yahoo were signatories to an open letter in early December on reforming Government surveillance (http://reformgovernmentsurveillance.com/), Cisco was not a signatory despite the acknowledged impact this is having on earnings.
What is Cisco doing to influence public policy in this area?
-Jim
Jim,
Hey! Good to hear from you, and I hope you are doing well. FYI, the first we learned about the Reform movement was when it came out in the press. We weren’t asked ahead of time, as best as I know, and that’s ok.
We’ve been huddling up internally on this topic in parallel to the technical measures – e.g. what do we want to say/do in the public policy arena and haven’t decided yet. More on that as we get clear about it, and I will definitely ensure our decisions/efforts get more visibility.
peace,
–jns
You are awesome, SpinDetector!
SpinDetector – Great comment! This denial does not help worried customers much.
//
Johan
Hi John,
You mention Ciscos “Vulnerability Disclosure policies”. Are those public available?
Thanks,
Johan
Johan
re: Vuln policy, I had added it to the bottom of the original post. Here it is again just in case…
http://www.cisco.com/web/about/security/psirt/security_vulnerability_policy.html
–jns
I have nothing against cisco, but I lost confidence in any american company … Stop working with this government …
I am here in Belgium. Yesterday I read “der Spiegel” and Fear now to use a computer ! Ask your president to stop playing his game … unless you want us to stop buying American products …
Jean Pierre HERVEG
Please allow me to say that your suggestion ‘ask your President’ is not quite holding water. Presidents come and go and it is the systems that are behind the individuals which cause the problems or bring the benefits. Just like people keep saying ‘Putin is ….’ There are millions of people in Russia, and even a Putin comes and goes. This mindset comes from the times of the personality/celebrity cults. Thatcherism did not die with her, and Stalinism was alive and well after Stalin was bumped off. This pinning an issue on one person doesn’t do the trick.
In the US, my interpretation is that their President protects secret.gov because they protect him. Some new development needs to happen if anything is to change.
On IT, every wall one man can build, another can tear down. If it’s on a computer, it can be syphoned off.
Adelaide_girl / Hereveg
I hope we’ve been really consistent with what we don’t do with this, or any other, Government in other articles, this blog, and our disclosure and vulns policies.
Adelaide_girl’s point about if its a computer rings true, which is actually what keeps us all working so hard on security issues inside Cisco (including acquisitions etc). If it’s a government, an adversary hacker, a criminal, whatever… we’re committed to building technology that is best defense Before, that can react During, and then can ensure quick recovery and forensics After.
I’m very happy to respond to the comments about the NSA, the USG, and the like – this is a critical topic. That said, our team works to stop/detect all adversaries no matter how “you” would define adversary.
peace, j
This all may be true and you have a right to your opinion, but I say “Innocent ’till proven guilty.” I’m giving Cisco the benefit of the doubt.
Mary
Thank you for that benefit. 70,000+ people who work here appreciate your support, and are committed to keeping your trust, the trust of our customers, and the support and trust of our shareholders.
–jns
Thank you, John. I support Cisco and I’m using Cisco products and am very happy with them.
Mary
Unfortunately, the best way for us to find out that Cisco was involved in Government spying was for you to folks to disclose it, against Court order, in threaten to liquidate down the company instead of participating with unwarranted government surveillance.
Now, there is absolutely zero way any customer can be sure that their Cisco equipment or software are secure from government overreach. The level of sophistication that the NSA has brought to bear is such that we can’t know that your security protocols are subtly breached, that there aren’t minor, almost insignificant implementation flaws in cryptographic processes that weaken encryption, or other subtle yet exploitable problems that you have allowed to go uncorrected.
The bare minimum that must be done to preserve Cisco’s integrity is to open source the entire stack, including device firmware, and allow customers to compile and install their own audited, vetted, verified stack. Even then, we can’t be sure that Cisco and the NSA are not co-operating to intercept the actual hardware and compromise devices on the most low chipset level.
Basically, everything you say from here on out is too little, too late.
Dan,
You’re bringing up a few good points.
We’re likely not going to open source everything (albeit we’re doing more of that), BUT to achieve your goal, we sure are open to inspection of all sorts on our products. Today, our products are scrutized by third-party labs in multiple countries, governments, and companies and customers all around the globe, and we’re open to even more for transparency purposes.
You bring up another problem that is troubling us too, and Matt Bishop at UC Davis just posted a followup blog to my Features, Bugs, and Backdoors:
Mine: https://blogs.cisco.com/security/features-bugs-and-backdoors-the-differences-how-language-can-be-misused-and-a-word-of-caution/
Matts: http://matts-words.blogspot.com/
You can’t prove a negative (my philosophy teachings showed me that) – which means if something subtle outside of our control or awareness is manipulated such as crypto, and we can’t find it, we have a problem. We’re exploring some other ideas there too.
BTW, more on crypto here just to keep that thread going
http://blogs.cisco.com/security/a-crypto-conversation-how-we-choose-algorithms
thanks for all the comments, I hope my response helps,
peace,
–jns
This isnt a legal court its a court of public opinion, which you are in the minority. This is a matter of the feds legally suppressing multiple companies from speaking out or not cooperating. I for one will always assume the feds guilty until proven innocent.
It may not be Cisco’s intent or wish but the feds don’t play fair and really don’t care if it hurts the companies mentioned.
I for one would leary any product manufactured under an American company until some reforms are made, even then who can you trust.
Spin Detector,
Tens of millions of dollars are you kidding? Evedently you don’t know Cisco has about 50 billion in cash and investments. Yah betray customer trust for 10s of millions…You really think the D-link you will be replacing your Cisco with is more secure. Let me know when the alarm goes off..
mrflorida
I appreciate the POV you shared.
It’s for exactly this reason that we’ve said in the past, and again in what started this blog, that we don’t work with any Government to weaken our products or insert backdoors.
Speaking the truth is easy; trusting it is true isn’t always.
–jns
John:
Cisco is under gag order NOT to reveal the filthy deals between Cisco and NSA, it is crystal clear to everyone now: all leading IT firms work with NSA, Cisco is probably the biggest culprit because you provide the backbone of the internet. Without your help and intentional weakening of your code, NSA will have a hard life. NSA has capable folks, so does Cisco.
The truth is this: Cisco is totally untrustworthy. Just wait for more dirty secrets to come out!
Actions speak louder than words and are now needed to rebuild trust.
You say you don’t work with any government, but that’s not the point. Neither did Google and their corporate infrastructure had been compromised.
You say:
“At this time, we do not know of any new product vulnerabilities, and will continue to pursue all avenues to determine if we need to address any new issues. If we learn of a security weakness in any of our products, we will immediately address it.”
So you have been aware of ‘Jetplow’ for Cisco ASA as this isn’t new information for you?
The US government claims to have low level malware for your security appliances and you do what? Wait for more prove? Ask the Spiegel for public available information? What’s next? Ask the NSA for handing you the exploit or truthful answers?
The fact alone, that you are only “concerned” the US government is developing, selling and running malware for your security devices is alarming.
The slide states:
“Status Released. Has been widely deployed…”
What are you doing now to keep your current customers of security devices safe?
Give your customers asap the ability to check the integrity of their security devices.
What did you expect, John here is just a low level spokesperson paid by Cisco to play dumb. It’s not like he’ll tell you the truth everyone knows, that Cisco is the biggest partner of NSA in the network world.
NSA takes good notes of what we said, and creates democratic files containing our references and all what we said … By the way, the bad ones are the Russians and the Chineses. NSA had nothing to do with KGB. NSA is made of good people, KGB of bad people … Obama is a good guy and Putin a bad one … One day I will ask to become American I have a good “basic knowledge” !
Christoph
Thanks for the blog post and questions. Clarity is essential here for you and me both.
No, we weren’t aware of JetPlow until it showed up in Der Spiegel. We have asked Der Spiegel for more information. We are still waiting to learn if they have any, or will share.
Here’s what we are doing internally. We’re trying to figure out what could have been done, based on the preso, and then figure out what – if anything – we have to fix.
We are literally *building* an integrity checking tool for ASA platforms right now, to give you what you and others are asking for to test your own devices. I noted in another blog post that there is an industry wide gap. Where integrity/forensics tools have been built for other OSs, none really have been been build for ours.
We used the word concerned because we knew so little, and frustratingly, haven’t gotten anything else to help us decipher if tools really exist or were desired, how they work, what OS versions on what platforms were they working on if any, who was targetted, etc. All of that would make this a heckuva lot easier. Ok, enough whining there.
Instead, we’re focusing on what we can control – investigating, theorizing, and then looking for weaknesses. We do this pretty regularly and now it’s an all hands on deck effort so that assuming we don’t get any more help from outside our company on what really happened, we work the problem inside.
I hope this answers your questions. More soon on the tool, our vuln search, and our action plan for anything else we learn.
–jns
John,
Thanks for engaging so intensively on this blog post.
My question is this: How can users of Cisco products have confidence that senior management are aware of the potentially significant actions being taken by employees with higher security clearances?
This relates directly to the quality of information a senior officer (such as yourself) could expect to have regarding interaction between Cisco and US intelligence agencies.
When discussing the Snowden revelations, it has been common for senior officers to attempt to bolster public trust by saying (as you did) ‘as an senior executive at Cisco, it better darn well be true if I say it’.
However, while this ‘seniority=comprehensive oversight’ equation holds true in most areas of corporate governance, the exception seems to be in highly sensitive matters of national security.
In a NY Times article it was noted that ‘Tech companies might have also denied knowledge of the full scope of cooperation with national security officials because employees whose job it is to comply with FISA requests are not allowed to discuss the details even with others at the company, and in some cases have national security clearance, according to both a former senior government official and a lawyer representing a technology company’ (7 June 2013, ‘Tech Companies Concede to Surveillance Program’).
I’d be most interested to get your thoughts on this quandary.
Many thanks,
Dave Clemente
@Dave_Clemente
The problem is mrflorida that there is no way to know that Cisco was not ordered to do so by a secret US intelligence court, then barred from disclosing the order.
This has happened to other US companies. Who have decided it was their duty to comply. One US CEO said to do otherwise was “treason”.
Dan
I think there is a way – which is for us to say “we’re not” (see another posting I did saying just that). It’s one thing not to disclose, it’s a wholy different thing to say “we’re not”…
thoughts?
–jns
John, this is valuable only to the extent that you are aware of the truth, and this is valuable only to the extent that you are permitted to answer honestly.
Unfortunately the nature of secret courts is that once suspicion is out there we can’t know what rulings you are bound by. Who is say, like the Lavabit case, that you not are being forced to maintain the government’s cover in in the investigation. Lavabit had to fight tooth and nail to disclose that they were under court order, and it nearly ended the proprietor in jail anyways.
Likewise, who is say that you are able to speak to everything that has happened in Cisco’s back yard for the last decade.
The subtlety with which the NSA is breaking the US tech industry is striking. A straight up backdoor is unlikely, those types of things are clumsy and easily found out.
But how can you say that the NSA did not lean on a group within Cisco to leave in a vulnerability in a random number generator, or to incorrectly offset hash calculations, or any other myraid of ways?
And, worse, is let’s say that Cisco really didn’t have any involvement in that. How can you ever prove it? The Trusted Devices is only so good as it can be verified all the way to the bottom turtles. Can you vouch all the way back to the fab that makes your silicon that the design is what has been produced? Can you verify your stack all the way to the source, to firmware and device level opcodes? Can a customer?
The real problems are deep:
1. Cisco cannot prove that they are not under secret court order to co-operate with an ongoing investigation into something that is under the jurisdiction of a secret US court.
2. Cisco cannot prove that you aren’t under a gag order forbidding you from disclosing such an order.
3. Cisco cannot prove that their entire employee, vendor, fabrication chain is secure for every device you have in the field today.
4. Cisco cannot prove that their entire software toolset is secure, from build tools to updates.
5. Cisco cannot prove that hardware they shipped from their manufacturing partners or facilities was not intercepted and modified by the NSA before reaching a customer. And, as a customer, I cannot prove that to be the case either.
So we all appreciate your claims and your flat denials, but they can’t help me know that my data and my clients data is secure. I cannot promise to my clients that our communications are as secure as I thought them to be two weeks ago.
While I reserve final judgment as to whether Cisco has been complicit in these and other newly-revealed activities, I do think that Cisco’s Security Response (Document ID: 37486; Revision 1.0) [http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131229-der-spiegel] is extremely misleading.
After noticing the Der Spiegel article and linking to it, the Security Respoonse says:
“The article does not discuss or disclose any Cisco product vulnerabilities.”
However, the second sentence of the second paragraph of the article clearly states:
“A document viewed by SPIEGEL resembling a product catalog reveals that an NSA division called ANT has burrowed its way into nearly all the security architecture made by the major players in the industry — including American global market leader Cisco and its Chinese competitor Huawei, but also producers of mass-market goods, such as US computer-maker Dell.”
While the article indeed does not “discuss or disclose” any SPECIFIC “Cisco product vulnerabilities”, the article does in fact “discuss or disclose” that UNSPECIFIED Cisco products DO in fact have security vulnerabilities known to the NSA.
While Cisco’s statement may simply be an unfortunate choice of words, it inevitably gives the impression that Cisco is either being parsimonius with the truth (at best) or actively misleading (at worst). In any case, Cisco’s statement does not inspire confidence.
I hope both that Der Spiegel publishes the documents in question and that Cisco subsequently publishes a satisfactory response following a comprehensive investigation.
Adam,
I see your point. The documents are dated from years prior, so we’re not sure at this stage if the vulns that are allegedly being used are already fixed, if there are new ones we don’t know about and need to fix, or what.
Our response maybe should have read:
“The article discusses Cisco products, but does not discuss or disclose any specific vulnerabilities (old or new), so we are seeking more information and will pass along what we learn.”
would that have come across better? thanks for the feedback, j
I see that just as I was posting, Cisco posted Revision 1.1 to its Security Response that now includes the following :
“On December 30, 2013, the German news publication Der Spiegel published additional information regarding the alleged creation of implants for some of the Cisco PIX and Cisco ASA platforms. The Cisco Product Security Incident Response Team (PSIRT) has opened an incident to investigate the situation. The incident ID is PSIRT-1384943056. We will communicate our findings through our standard security disclosure process.”
Unfortunately no link to this subsequent publication is provided.
(Moreover, the revision still includes the mis-characterization of the original article.)
Who do these American “secret” agencies think they are.
If I had to do something remotely similar to this, I’d get thrown into jail and labelled a “cyber-terrorist”.
I challenge all the companies violated by this:
a) to investigate it
b) to sue the perpotrators
c) release fixes / patches / detectors / etc.
Show us how serious you are then we will take you seriously,again.
Hello,
I was wondering why are you not considering to use a simple & verifiable register (a HW independent chip – a ROM (very important to be a ROM) with an user input & checksum routine and an EEPROM for the checksum itself) for the future generations, in order to store the SHA checksum of the Firmware / IOS currently running on the CISCO machines (same for updates), so that any further tampering of the images would cause the machine not to boot / stop working if already running. Pretty simple and cheap safeguard. But maybe such a safeguard should have been already implemented by now, especially when it comes to such a serious company, with so much responsibility (so many networks relying on your HW). By the way, are you not looking for some serious people at your R&D department? Never mind…
Have a nice day!
Just a Guy
We agree with you, which is how the Trustworthy Systems initiatives were born.
Here’s the thing, we have many, many products and you’re talking about hardware modifications. This means two things. For new products, we can incorporating hardware anchors, advanced memory protection, detection against manipulation, etc. all from the beginning. This is all referenced in the Trustworthy Systems website, and is industry bar setting (I think).
For existing products, it means revving hardware and versions and we’re doing much the same thing as with new products, it just means re-tooling a bit internally (which we’re doing quickly, and probably never fast enough, but…)
The last area which needs more work is forensics. There is very little commercially developed in this area for routers/ firewalls/etc, either by us or other companies that might specialize in that domain. More to do there.
–jns
The only way US IT firms could regain trust is to disclose everything you had with US government, which I know it is not possible because it is so dirty!
I tried to buy a router a month ago, I avoided the Cisco brand even it is discounted. Wait for Europe to slowly move away from your products (Swiss TV RSI said that FINMA suspect US government intercept all Swiss banking information, and the stupid FATCA is going to be voted down by referendum, just remind you this: Switzerland had always been an admirer of the US, no more!) , your company is following the way Motorola progressed in the last decade, feel bad for your employees, because most of them are just trying to make a living!
I know NSA is monitoring my computer by hacking into my computer, but anyway, the only way Cisco can clear its name is to move off the USA and dump its US business if deemed necessary, otherwise, you will NEVER gain trust. Because you are under gag order not to disclose your dirty secrets, that only destroys your trust.
I personally knew a guy whose father installed hackware onto things sold to Warsaw block over 30 years ago. You guys are doing the same thing. No doubt about it (why does Cisco and Intel has dept where top security clearance is required???)!
I have little problem for NSA to spy on foreign government, but I have a problem when NSA spy on the innocent people and tell them they have no expectation of privacy (this hurt US economy, which is far more important than the interest of a small segment of society: human-killing tool manufacturing business), you know what, that is fine, but they will and they should boycott US products, also I have a problem when the whole IT firms sold their trust to government, now no one trust them!
The US based IT firms will NEVER gain trust from outside the USA. Another thing: did Cisco just lose a significant business in the core router segment?
Hello John
After going through the NSA documents for cisco product like ASA FW and router and other IT company product,I wonder how NSA can implants the PBD into your BIOS,Do you have any potential vulnerbility identified now or may show some evidence on implants entry comes from upstream supply chain or public recognised routing protocol
Bob Syln
Bob,
thanks for the question. At the moment, we don’t know if/how this was achieved albeit we have a lot of theories. In another reply to a comment from “Just a Guy” I was talking about the work we’re doing on hardware anchors, validated software/signed images, and a bit around forensics.
The information we’ve received to date doesn’t give us much. From what we’ve been given, mostly through the press as you’ve seen as well as I have, doesn’t talk about unknown vulnerabilties, or if the capabilities were using older and fixed vulnerabilities (noting the documents published are a few years old), etc.
So we can control what we can control – which is to do our own homework and see if there is something we’ve missed that we can find, work it, fix it, and then tell you all about it. That’s what we’re doing now… which is something we do every day btw with a really good internal team that breaks our own products so we can fix them.
–jns
Should we stop buying American products …. ?
I don’t think so – nor do I think American is even an appropriate way to describe Cisco’s products. We develop in many countries, manufacture in multiple countries, and have business offices/employees in many countries. Ok, I know what you’re going to say, I’m just offering up a POV that says we really think of ourselves as a global company also.
We are a German IT company specialized on IT security and IT fornesics.
Due to our special history (Third Reich / Stasi) we have a more serious view to the exploitation of civil rights: in Germany the actions of the NSA are a complete NOGO.
Since June we’re advising our clients – if possible – not to buy any American IT products any longer. Our global player – Deutsche Telekom AG – is planning to build an own network inside Germany. We have very good products ‘Made in Germany’ they should be more safe from influence of NSA.
I think that’s the beginning of the end from the famous U.S. IT companies. In the near future we’ll separate the Europe Union from the United States and from the crappy United Kingdoms with its GCHQ.
Lots of Americans seem to be too stupid or to less smart enough to see the madness the NSA is doing. The long they can pay all with their VISA or AmEX all seems to be o.k. How stupid!
It’s time to say good-bye to the United States and to the United Kingdom.
That’s interesting, let’s stay european … I will let them play their own patriot game
Michael, you’re embarrasing. Your “We are right, They are stupid” monkey attitude is dumb enough to actually ignore you. But you are bold to make your words all German’s words, so as a German I need to respond to your idiot’s crying. You have learnt nothing from history, otherwise you wouldn’t write that disrespectfully, so talk for yourself, only, or shut up.
In fact, NSA’s and GCHQs mass surveillance affect civilians all over the world. With such tight cooperations between NSA and GCHQ, for example, the latter could easily do the dirty work on behalf of US civilians helping NSA to bypass US law.
Regarding Cisco’s role: What I can not draw from Mr. Stewart’s answers – and that worries me deeply – is, that he is not even considering legal measures against NSA and US Government as a result of those revelations. That is quite striking, considering that Cisco’s credibility actually has been (or is about to being) destroyed and even more striking for a company being a manufacturer of fundamental parts of the WWW-infrastructure. So I agree with folks who ask Cisco to become aware of their social responsibility, now!
NSA is a cancer. We’ve nothing against Cisco but my company stopped buying american IT products.
Peter
I’m glad you have nothing against us… I’d like to point out that, and maybe this doesn’t help I don’t know, but our Engineering team is comprised of people all around the global from Poland, to Prague, from China to India, from the US surely, and multiple other countries. Yes we are headquartered in America, but we’re not solely an American engineering organization, test organization, etc.
just thought I’d add that as food for thought. –jns
As ex-Cisco employee, I know Cisco has a secret department to handle sensitive stuff, am I suggesting they are actually planting spyware onto all the hardware sold to countries like China, Russia, India, Arab countries?
Of course I don’t have 100% proof, but people will never TRUST this unethical company, PERIOD!
I would be inclined to believe that cisco would not have knowingly allowed the NSA to implement backdoors and helped to write code allowing siphoning of data without the admin’s knowledge for a simple reason.
respect.
you can’t buy it, you have to earn it and its the most valuble comodity in the marketplace.
look at the other major companies whose business models revolve around advertising: facebook, google, yahoo.
its in thier interests and normal operating manners to “sell” your data to advertisers, so why not sell to the government?
but now look at cisco, its business is to design and produce network equipment, it doesnt derive its primary revenue stream from advertising.
selling customers data to advertising agencies doesnt gain cisco much money and erodes brand respect, so theres no point, it can gain more revenue and customers by being trusted and respected.
GCHQ and the NSA are spy agencies, what did you expect?
this is what they do, its their whole purpose, to capture data and analyse it to catch criminals/terrorists/rogue governments etc.
are you also naive enough to think your countries (whereever you are from) doesnt have a department/agency dedicated to domestic intelligence that isnt so well heard of, but just as capable of breaching your privacy?
the ways by which these agencies have used technical knowledge is natural to expect and the technical methods demonstrated are exceptional, the problem is the apparent lack of political oversight and control applied to the agencies in how/what/when the techniques can be used and the overstepping of the remit of the agencies into foreign “friendly governments”
“Since June we’re advising our clients – if possible – not to buy any American IT products any longer.”
This is an example of security theatre. It is obviously not going to make you any more secure. The NSA is attacking products from non-US companies as well. The hacking community has been demonstrating for decades that you can backdoor software and hardware devices without the help of the vendor. There is no indication that the US companies are complicit in these attacks any more than the non-US companies are complicit. Firefox has been attacked by the NSA in the same way. Are you going to advise people to stop using Firefox as well?
“… There is no indication that the US companies are complicit in these attack … “: they are complicit, most of them ! Not buying American products is a method to force these companies to act (they are only interested by money). In addition we should make local networks.
We’ve said we’re not complicit, and are fighting to keep your business to include this blog. I, and all of us here at Cisco, hope you will change your mind and stick with us and by us.
–jns
Here at Swisscom in Lugano, Switzerland, we had a conference back in October, about securing our network and improve performance because our customers are overwhelmed by oversea request to have data/cloud service, because they trust Switzerland, not Amazon . I talked some data center companies, they business went up 300% since June.
Also the company is working much closer with European suppliers, even we are keenly aware NSA can penetrate their products as well, we hiring more security professionals (you can call them crackers, hackers) to improve our security, we will reduce our orders from US IT firms, for sure!
it would be helpful to all concerned if the IT industry would organize to lobby congress to rescind the “national security” letters that prohibit exposure of secret (and illegal) agreements to provide govt access to private networks and hardware.
Cisco stepped forward in response to the latest NSA bombshell. Cisco, for its part, posted a blog post Sunday, saying it was “deeply concerned” by the report’s findings and that it “will continue to pursue all avenues to determine if we need to address any new issues.”
“If we learn of a security weakness in any of our products, we will immediately address it,” wrote John Stewart, senior vice president and chief security officer at Cisco.
Better confess now, if Snowden force you to confess, you can be Cisco will be boycotted worldwide. I believe this punk will reveal more damaging stuff. Cisco is doomed because of your devious practice against Chinese companies. How r u doing there?
I think real punks are the NSA people. Here outside “your country”, Alexande, Clappers look like real gangster… Snowden tells us where gangsters are and what they are trying to do. Now what you have to do is restore confidence !
a Belgian Citizen.
If you child visit a doctor, and the doctor put botulism virus in his body, after you discover this, are there anything the doctor could do to restore your trust in him?
My guess is no.
The distrust of Cisco, and all US based IT firms will spread like wild fire.
Due to the creepy nature of secrecy, there would be no restoration of trust in US based firms, and I feel sorry for many many people employed in this field!
You’re right, the problem here is due to the numerous lies. They start lying in 2001, showing us on TV mobile structures used by Saddam Hussein to built atomic bombs… and they are still lying now … No one cab believe them ! They surely think we are stupid !
“Cisco is under no secret US government orders as of the time that I write this (nor were we at any time beforehand to the best of my knowledge and that covers 11yrs here), nor has the US government (or any other government for that matter) compelled us to secretely do something to our products to weaken them in any way.”
I am pleased to read this statement, but I wonder about other ways that Cisco has helped the government subvert its products.
For example, has Cisco provided source code for its products to NSA (or the State Dept, or the Commerce Dept, both of which shill for NSA) in order to get export permission?
NSA doesn’t ask for your source code “merely” to make sure that you have a copy of it. They use it to exploit your product! For example, if their favorite malware gets burned into the BIOS, it certainly would help them to get a copy of the BIOS source code and the code for reflashing the BIOS. Can any customer reflash the BIOS with an image of their choice? What “security by obscurity” do your products use to prevent customers from modifying and reflashing the BIOS? You probably told NSA all of that — you were probably required to, in order to get permission to export your products from the US to other countries.
This doesn’t qualify as “compelled us to secretly do something TO OUR PRODUCTS to weaken them in any way”, because you didn’t change the products, you just gave a bunch of info to the federal government that you don’t give to the public or to the vast majority of customers. But it does qualify as “a very strong federal requirement to secretly do something to weaken your products against the NSA”. You are not COMPELLED to ship products outside the US — you could throw away 60% of your business — but if you do ship them, you are compelled to help NSA break into them by privately (secretly) providing information to NSA.
John, excellent points, I have noticed that all the rebuttals from IT firms are carefully worded, including the response from RSA. Why is that? Looks to me they are issuing no-denial type denials!
Trust is the most important/fragile aspect of human beings, in old Asian traditions (>100y ago), you simply don’t betray your love ones and close friends. If I ever commit perfidy, the only thing to wash off all the deserved distrust is to use my blood. Another way to say it is: I would rather die instead of betraying my love ones and those close to me.
There is a fiduciary responsibility for companies to ensure they act in good conscience. These who don’t will not prosper in short term, and likely will not survive in the long run.
John,
My apologies again for the delay in responding. I somehow didn’t see this post and found it after your followup.
As I noted in all of my responses, and in response to your follow up question (thanks for posting that btw) – we don’t help the U.S. government (or any government) to deliberately weaken or compromise our products. This includes what might be considered indirect ways.
About the premise behind your question: that Cisco cannot export products, unless it hands over its source code to a USG agency. We are not required to hand over our source code to anyone for export-related reasons, actually, nor did we. We are under no requirement to provide source code to the NSA (or another agency connected) to secure export approval. U.S. export controls are about ensuring specific technologies are not available to unauthorized countries or organizations. We definitely have to file a lot of paperwork (and that’s not much fun btw), and sometimes talk to representatives of organizations, to gain permission, but we don’t need to hand the code over, or give access to it.
For Cisco, more information on these technologies and restricted export locations can be found here:
http://www.cisco.com/web/about/doing_business/legal/global_export_trade/deemed_exports/controlled_technology.html
You correctly noted that Cisco has many customers outside the United States. In the first quarter of Cisco’s FY14, for example, customers outside the Americas contributed 39% of our revenue. The remaining 61% included U.S. customers, but also customers of other North and South American countries. Given this global profile, we clearly know how important non-U.S. customers are to our overall business. And just like our U.S.-based customers, these organizations rely on the integrity of Cisco’s products and us as a trustworthy vendor. We know that, and take it with the utmost seriousness.
So I feel like a broken record, and am running out ways to say this and only hope that our back and forth is helping: Cisco does not work with any government to weaken our products, nor are we compelled to do so for any reason. Our interest is in supporting our customers (regardless of where they are located) and continuing to strengthen our products against any sort of compromise.
hope this helps, and again, sorry for missing the original post. Good question.
-jns
John, what means will you provide for customers to verify the integrity of their equipment?
I challenge Cisco to let third-party to verify the integerity of their products, of course Cisco will say no because of trade secrets, blah blah. In the end, there is absolutely ZERO trust in your company, in your product. There will be a silent boycott of your products world-wide, same thing would happen to RSA and its parent company EMC. We are acutely aware this may take a long time, but we are also 100000000% determined TO UNDERMINE companies who sell us poinson, such as RSA!
I sensed that there would be more leaks coming out to refute these “innocent” rebuttals issued by other tech companies!
Christoph
Thanks for asking. We’ve found a gap – both outside and inside the company. There are no real “forensic” tools in the industry for routers/etc and we’re prototyping a bunch now. More as fast as we can, because we’re literally building the toolsuite on the fly.
–jns
All the stuff is here
http://leaksource.wordpress.com/2013/12/30/nsas-ant-division-catalog-of-exploits-for-nearly-every-major-software-hardware-firmware/
Yes, but will you make these updates available for EOL and EOS products? How about equipment owners that no longer have support contracts?
This is an issue where the NSA is flashing the loader of legacy/current Cisco security products.
If I have a functioning PIX that was “rooted” and is EOL – Cisco should be issuing updates for these units! They are clearly defective regardless of the EULA b.s.
The delectable irony.
Huawei being accused of being a subvert chinese spy and having to jump through hoops to certify equipment which are mostly blacklisted at the onset of mere mention for American markets.
Cisco being an American darling selling “edge network” certified equipment worldwide (including china’s great firewall) having their whole pix firewall line compromised under code name “JETPLOW” working in conjunction with “BANANAGLEE” software implant. Total unit cost of “$0” per NSA slides for marketing.
Customers will put a higher cost than NSA’s Trojan inside sales discount
Our clients panicked recently, demanding to know if there stuff is secure? My answer to them:” As secure as NSA wanted because we used Cisco products”, and the customers were infuriated, and demand me to do better, so this is what I am going to do:
I will have no choice but to boycott Cisco and Juniper, I will buy from Nokia, and even Huawei (I can’t imagine they dare to put any spyware because NSA will uncover it in no time and reveal to the whole world in nano second, I suspect US hate Huawei so much because US can’t ask them to install spyware in their products, otherwise the secret program will be known to Chinese). We are aggressively looking for alternatives, also we are going to commit more to ARM based servers, thinking it is more open than x86, so we have a better control!
Al Qaeda is finally successful in destroying America: both in freedom and Us economy. Cisco, you are accomplice to their evil/despicable goals.
Lets get something straight. I understand now, after my purchase, that hardware and spyware are subvertly added to computers. Well I got a bone to pick with the manufacturers.
Product Liability. Is this hardware and software tested and proven to not affect the utility and lifespan of the computer? I want to see these test results.
Warranty. Does any of the spyware hardware added subvertly post manufacture by third parties with knowledge of the manufacturer compromise or void the warranty? Are computer failures linked in any way to the post production installation of the subvert hardware and spyware? Is there documentation proving one way or another?
Contractual Consideration. Every agreement requires consideration. I looked through all my documentation and there is nothing indicating agreement to install post production spyware or hardware or the consequences. Manufacturers are paid by government agencies to allow these subversive spyware and hardware installations. Does the cost of computers include the costs of this undisclosed, post manufacture modification? Why do I have to pay for post production changes? Why wasn’t I informed?
Legal. This must be kicked down to the wolves in legal. Given the items I listed above, as a consumer, I am offended that I am stuck with the costs of this post production modification, my warranty must certainly be compromised, the effects on unit failure are in question, and I do not see contractual consideration.
Manufacturers that knowingly allowed, were aware, or involved in post production installation of spyware or hardware without disclosure to the consumer are liable for the costs and therefore consequences of recourse.
Choice to manufacturers. Remove, replace, or refund. If you cooperate with spies for profit, I better be notified before the purchase of any consequences resulting from post production installation of spyware or hardware in writing.
Legal. This is the biggest product manufacturer lawsuit in history handed to you on a plate. Manufacturers are responsible, manufacturers are liable. Lets see what 250 million lawsuits and recalls do for the thought process. Sick Em.
As a former Cisco distributor, I am furious for your devious act of fabricating allegations and kick out Huawei, so you can monopolize the US market.
Now your dirty secrets coming out, the whole world should boycott your products and drive your company out of business. (You produce nothing that can’t be replaced by at least two other companies)
There are many honest IT business (mainly small one), who are unfairly hurt by criminals like John Chambers, and lead to job losses. Most Cisco employees are just honest Americans, and I have nothing but respect for them, but for a few top brasses, who are evil beyond words, when you die, you will face god, and god will ask to you to burn in hell!
Open source is the future, and hopefully US can still lead, but evil ones like you and Microsoft, have no place and should be boycotted by everyone!
John Stewart, your statement is full of lawyer-drifted double speaking weasel that allow you to play dumb and lie without getting into legal trouble. And that’s what you’ve been paid to do.
You think your customers are stupid don’t you.
Nope, I don’t. Quite the contrary really. I think our customers are really smart, having met quite a few of them.
–jns
“At this time, we do not know of any new product vulnerabilities”
Stop pretending to be innocent, these words no longer carry any meaning after Snowden.
If I were Cisco CEO I’d order you to keep your mouth shut.
http://news.techeye.net/business/huawei-products-do-have-backdoors
As a former Cisco distributor, I am furious for your devious act of fabricating allegations and kick out Huawei, so you can monopolize the US market.
Now your dirty secrets coming out, the whole world should boycott your products and drive your company out of business. (You produce nothing that can’t be replaced by at least two other companies)
There are many honest IT business (mainly small one), who are unfairly hurt by criminals like John Chambers, and lead to job losses. Most Cisco employees are just honest Americans, and I have nothing but respect for them, but for a few top brasses, who are evil beyond words, when you die, you will face god, and god will ask to you to burn in hell!
Open source is the future, and hopefully US can still lead, but evil ones like you and Microsoft, have no place and should be boycotted by everyone!
ohn Stewart, your statement is full of lawyer-drifted double speaking weasel that allow you to play dumb and lie without getting into legal trouble. And that’s what you’ve been paid to do.
You think your customers are stupid don’t you.
We have stopped buying Cisco products. It doesn’t matter if NSA has hacked Cisco and has placed backdoors without the management of Cisco knowing it or not. It doesn’t matter if its a backdoor or if NSA just found a way to hack into Cisco. All what matters is that Cisco products can by no means be considered secure today. And even if cisco fixes all this, the threat remains because the US fucked up law allows the governmental agencies to ask Cisco in the future to put backdoors back in officially so wiretapping without warrant will be possible again. Something which is totally unthinkable in Europe.
It’s not only a issue of Cisco. It’s an issue for all US based infrastructure and service providers. You just joined the club of blacklisted vendors. Huawei and Juniper is there too. Your fault might only be that your company is located in the wrong country. The country which is constantly at war with the world, where peace is an unknown word, where weapons are now being manufactured by Silicon Valley companies.
Just watch this http://boingboing.net/2013/12/31/jacob-appelbaums-must-watch.html?utm_source=dlvr.it&utm_medium=twitter
John,
Thanks for engaging so intensively on this blog post.
My question is this: How can users of Cisco products have confidence that senior management are aware of the potentially significant actions being taken by employees with higher security clearances?
This relates directly to the quality of information a senior officer (such as yourself) could expect to have regarding interaction between Cisco and US intelligence agencies.
When discussing the Snowden revelations, it has been common for senior officers to attempt to bolster public trust by saying (as you did) ‘as an senior executive at Cisco, it better darn well be true if I say it’.
However, while this ‘seniority=comprehensive oversight’ equation holds true in most areas of corporate governance, the exception seems to be in highly sensitive matters of national security.
In a NY Times article it was noted that ‘Tech companies might have also denied knowledge of the full scope of cooperation with national security officials because employees whose job it is to comply with FISA requests are not allowed to discuss the details even with others at the company, and in some cases have national security clearance, according to both a former senior government official and a lawyer representing a technology company’ (7 June 2013, ‘Tech Companies Concede to Surveillance Program’).
I’d be most interested to get your thoughts on this quandary.
Many thanks,
Dave Clemente
@Dave_Clemente
Dave,
Most welcome and thanks for continuing to ask questions. Let’s broaden your question even further than just national security: how can we be 100% sure of the potentially significant actions of employees that happen without our knowledge or permission?
For example, if someone is angry and works here (insider threat) how can we detect if they are changing code in such a way that would create risk for you? This is a huge industry dilemma in security, supply chain, software development, you name it. It doesn’t matter what nationality you are, or where you are incorporated, or which customer. So, you do what is known to be effective, and its the reason we started the Cisco Secure Development Lifecycle and Trustworthy efforts, and also the Secure Supply Chain work. It’s about having multiple checks along the way while we’re building a product, and then being in plain sight for all the world to find problems we’ve not found, and then handling the problems professionally to fix it. We strive to do all three.
Speaking of multiple checks, all our customers (including you I hope), researchers in the industry (special call out to FX here), and vulnerability vendors pound on our products to find problems too. What we’ve found as the best way to lower the probability, whatever the reason, of something happening without us knowing it is the checks/balances system and test-test-test.
hope that helps, and on the employee-could-do-it-but-management-not-know-front, I’ve asked everyone and I’m quite sure that one Cisco exec (of only a few) would have had to agree and none of them did.
–jns
From my point of view, there is a growing need for a massive public statement regarding the allegation against Cisco.
Either cisco collaborated or they did not.
Now is the time to say the truth!
Cisco may well tell the truth that about them not inserting backdoors. But a backdoor can also be automatically inserted by the compiler (or any other part of the toolchain) upon finding a particular token (string of characters, comment, etc.) in the source-code. So the source code can be completely clean; the only way to find out would be to decompile and compare.
And then check the compiler? It may be identically rigged. Its sourcecode can be clean, but every compiler that compiles its successor is not to be trusted. Also, the linker is also not to be trusted…
John,
There is a way forward, though it is not without cost.
Secrecy can only be allowed when there is trust that it will not be abused. We now find that it has been abused and that that trust has been violated. No more secrets.
Separate from the secret community entirely. If you continue to do business with the secret agencies, you will lose your trustworthiness and, with it, your international business. No more secrets.
Cisco has (as mentioned earlier in this discussion) a department where employment requires US government security clearance. Close that department, abandon that business, and make holding or ever having held a government security clearance (from any government, not just USA) a disqualification for working at Cisco. No more secrets.
Cisco is a multi-billion-dollar-a-year company. You have influence on Capitol Hill. Use that influence to get laws passed retroactively voiding any and all NSA/CIA/FISA/FBI/etc. gag orders. Even better, get mandatory disclosure passed, with failure to come clean on the part of anyone who has received such an order made a felony. No more secrets.
Where is your value proposition? Is IOS what Cisco sells, or does Cisco sell network gear that happens to use IOS? If the latter, look very, very hard at releasing IOS as Open Source/Free Software/etc. The ongoing NSA scandal is destroying trust in proprietary software of all types. No more secrets.
If worst comes to worst, and the government is so arrogantly out-of-control as to impede the above measures, you must take the nuclear option: vote with your feet and leave the country. If the government prevents Cisco from acting to regain our trust, then Cisco must move to some country that respects basic human rights. No more secrets.
The bad ship USS SHADOWTYRANNY is sinking. You are not the captain. Choose: will you go down with it anyway?
news flash: Our company is panicking already, on Monday we have a meeting to discuss possible dumping of all cisco devices. Obviously, its going to take a while like a year or so.
I’m afraid the suspicion is getting real:Backdoor found in Linksys routers – https://github.com/elvanderb/TCP-32764
I’m curious to hear what Cisco has to say about this new evidence…
I criticized Cisco two days ago, now someone implanted back door on my PC, it freezes in no time after rebooting, since I’m experienced SA, so I know this comes from outside, ( is that Cisco? or NSA?) I reinstalled by using clean backup, same issue, The whole world should see my post, and Cisco is truly evil. Even when my computer freeze, hard drive spin like crazy, and light on the router flashing crazily, meaning someone transferring data to outside.
Yes, I know this post will be erased soon, but I am video taping my PC problem!
Sigh. Another evasive, overly specific denial. Let me ask for a real denial: is Cisco aware in any way of the NSA’s ability to compromise Cisco’s products? This wouldn’t be a ‘new vulnerability’ nor would it require Cisco to ‘work with’ any government agency. All it would take is not patching an old vulnerability as long as it seemed like only the government knew about it.
Jeff,
Simple question. Simple answer: no. We abide by our own disclosure policy.
http://www.cisco.com/web/about/security/psirt/security_vulnerability_policy.html
–jns
Companies should be indépendant of the law of the country in which they are. They should be free. Actually, if you have à factory in Andorra, you depend of the laws in this country. By the may most people don’t even know where it is, but it existed.
The NSA leaks and TAO tactics just go to show that all companies that operate in today’s digitally connected world should understand the basic fundamentals of IT, hacking and security implications in order to protect personal information. If you want censorship learn how to combat infiltrations.
On Thursday, January 4th in 2014, LM Ercisson’s corporate office has been bombarded with resuests (from European customers) to sell comparable products currently being provided by Juniper and Cisco, there has been a panic among European customers related to the fear of NSA!
Game over, Cisco!
I’m sorry, but as long as the Communications Assistance for Law Enforcement Act (CALEA), is still in effect. Not to mention the Secret FISA Court and their National Security Letters, such as the one sent to Lavabit demanding they hand over their private SSL/TLS RSA key from their servers.
I just can’t bring myself to be gullible enough to believe the US Gov isn’t mandating “surveillance” capabilities from one of the largest telecommuniction manufactures on this planet.
https://en.wikipedia.org/wiki/Communications_Assistance_for_Law_Enforcement_Act
I intend to voice my opinion on this matter by not buying any more Cisco security products until they come clean about the backroom deal they did with Uncle Sam.
Anyone who has completed an SF-86 in the last 5 years and worked more than 10 minutes in intelligence can validate several of the NSA’s claims. The fact is that some of the exploits are not possible without collusion only provides further validation to the claims.
Most of the security industry is run by people two steps away from wearing tin foil hats, and they want us to believe that collusion is a stretch of the imagination. Good luck with that…
Why haven’t they addressed this yet?
http://arstechnica.com/security/2013/12/nsas-broken-dual_ec-random-number-generator-has-a-fatal-bug-in-openssl/
I’d like to see another thread on this topic as well.
There is a Cisco Security Respons regardig that:
http://tools.cisco.com/security/center/content/CiscoSecurityResponse/cisco-sr-20131016-ec-drbg
Hello John,
if we can believe that no products of Cisco has been compromised by the NSA then we know that something is wrong with that intelligence agency.
Can you trust all your employees? Do you know how the NSA operates to get things done?
Well, I can assure you Cisco products are not working properly.
Either the hardware design and/or the production has been compromised.
I bet $10.000,-
You can’t tell us the truth about secret court orders to work with the NSA togehter. We can’t prove it if it is correctg what your are saying and you can’t prove it telling us the truth.
Like Google, Apple, Facebook and Microsoft lied in the first place I don’t trust Cisco products anymore.
Actually I don’t any US based company anymore because of this Hitler-Nazi-Secret-Court on US soil.
Hi John,
I have just got back from company’s meeting and it seems we shall stop using Cisco in few months. It is a shame, I had great experience with you. But it is probably a good solution in light of this scandal; we have no use for compromised security products.
By the way, your response is strikingly similar to Google’s Larry Page’s initial response to PRISM – he blatantly denied allegations; turned out he had to.
Cisco is a US-based company, the very large one, why would NSA leave you alone if they have already compromised Microsoft, Facebook, Google and now probably Western Digital, Dell, … Facebook is also here in Europe (Ireland), so being an international company did not prevent it from submitting to NSA. Why would Your case be any different? It does not make sense.
Good luck anyway!
T.
I believe in Cisco’s core values. Good that you have addressed this allegation.
thank you Cisco
Parca,
You’re most welcome. I hope this blog helps.
respectfully,
–jns
Good buy American soft- & hardware companies!
Welcome European soft- & hardware companies!
This may be an obvious question but in the wake of such a strong denial, how can Cisco possibly have complied with CALEA all of these years?
Hi,
I appreciate that Cisco tries to act openly on this topic. It is for sure no fun these days. Again, to strengthen it: appreciated.
I bought Cisco routers for many years mainly for two good reasons: they worked a lot better than the others I tried, and there was a certain level of trust that you know what you’re doing.
But as I had to learn about secret courts and gag orders and stuff that I didn’t know about before, things have for sure changed. Now I know I cannot trust, whatever you say, right? To be more precise: whatever an American company says.[1]
At the moment I see only three options out of this game with very uncomfortable rules:
1) Cisco open sources all of the material that’s needed to build a router so that security experts and coders can independently judge the hardware and software. Obviously that’s very hard for you to do, because that’s your asset. And even if it would happen, it will take a very long time. This is unrealistic, I know.
2) Cisco moves its headquarters and everything in a country without secret courts, gag orders and other stuff that makes it impossible to build trust. Not very realistic, too …
3) I buy hardware produced in Europe and put BSD or Linux on it. Not what I wanted, not 100% trustworthy, but a lot more than the current situation with Cisco and Juniper.
As I see it I don’t have a lot of choices because you don’t have a lot of choices. Bummer. I loved my MGS, AGS, 2000, 3000, 2503, 3640, 836, 871, 3745, 3662, 7509 and all the others I’ve forgotten. But if I can’t trust, then I can buy cheap stuff I can’t trust too.
[1] I thought a while before I wrote that sentence because that could mislead some readers that I’m anti American. No, I’m not. But that’s what I had to learn about gag orders, Lavabit and all the other companies from the US in the last half a year.
Best wishes that Cisco finds a way out of this mess.
Regards
Götz
I work for a large Japanese company, I have been told by my boss not to buy network gears from Cisco/Juniper/Alcatel-Lucent anymore, fortunately, such equipments are being built by Japanese companies, so we will buy Japanese equipment in the future (60% of our network equipment are from the USA).
Good luck to you, Cisco!
Quote: ” I have been told by my boss not to buy network gears from Cisco…”.
Well done Yamashita Atsuo and congratulation to have a boss who is able to use his brain.
I believe its inevitable that NSA would have used the biggest networking vendor (Cisco) for their “Tailored Access Operations”. The worst thing is that Cisco gets all this bad publicity which brings a big hit on the trusted image of the organization. I hope Cisco will “recover” soon from all of this.
Here is a Cisco published document for a Cisco data collector product. On Page 12, there is a list of default logins for the product. Above admin, with near root access (linux bash) is an account called “nsalogin”…. does this show that Cisco is cooperating with the NSA by building default accounts for them? http://www.cisco.com/en/US/docs/net_mgmt/smart_portal/Common_Services_Platform_Collector_Quick_Start_Guide.pdf
nsalogin represents Network System Administrator login, it has nothing to do with NSA!
You should absolutely have zero reservation about trusting Cisco and all other US firms (100000000000000x more trustworthy than your mother), they protect your privacy from China, Russia, NSA and UK, there is no backdoor in any US sourced products, not even in RSA products!
If you don’t trust, you are screwed because only the US produces such gears! I am buying Cisco shares like crazy. Their sales will triple in the next 12 months
DerekM,
Thanks for your post. My name is Russell Smoak and I look after the PSIRT team for Cisco which is responsible for managing the investigation and resolution of vulnerability reports in Cisco products and services. John Stewart asked me to look into your comment for the blog.
Cisco does not partner with any government or other organization to insert backdoors or weaken products. The username ‘nsalogin’ was chosen in 1998 when Cisco Advanced Services customers were known as ‘Network Supported Accounts’ (NSA). While an unintentional poor choice especially given current events, the nsalogin userid is one of 6 default accounts that ship with the appliance and is used for customer designated support resources to login to the collector. This account has no correlation to the United States National Security Agency. Cisco documents all default accounts in our product as part of our secure development procedures. The product documentation clearly outlines the privilege level for each account and how to change the passwords. The customer controls access policies and passwords on the device once it is deployed.
Because of the understandable misperception, Cisco is filing enhancement requests to change the name for this account. This is documented in the Cisco bug toolkit under (CSCum44128).
Thanks
Russ
http://www.cisco.com/security
Thanks Russ — my account team recently replied to me about this …. CSCum44128 was quoted…. I appreciate the reply.
Its simple like Jupitor said.
As long as the Communications Assistance for Law Enforcement Act (CALEA) is like the secret FISA Court and their National Security Letters are in effect, no one should trust any US-Company any more.
I would rather buy some RU-Products than US-Products until now.
The last weeks i was very busy in replacing all external Network-Products made in US with OpenSource-Products or European Products. Seems to be not so comfortable in some points, but its reasonable working…
Have Fun in a BigBrother/Stasi-Country, Norbert
Hello Mr. Stewart,
I want to thank you for your willingness to discuss this topic openly.
Still, since you may be under the threat of being legally bound to not disclose any cooperation with the NSA, it is just not possible to trust you right now.
I work at the IT of a large German corporation with 300.000 employees, and we are considering our options to move away from Cisco products.
Right now we don’t trust anyone. If there are confidential messages to be sent, right now they get sent by messenger (human person) with a suitcase chained to his wrist.
Hello Mr. Stewart,
I want to thank you for openly discussing this topic.
The dilemma is, we can not trust you nor any other American IT company right now, since you may be legally bound to deny any connection to the NSA.
I am working for a large German corporation with more than 300.000 employees and offices in most European countries.
We are considering our options to move away from Cisco hardware.
Our complete IT-Stack may be compromised and untrustworthy.
You will understand that this is a very uncomfortable situation for a company that is creating a lot of patents and technical papers.
To point out how serious this situation is taken: If there is a highly confidential message to be sent to someone outside the company, the message is *not* stored on a computer with access to the network.
The message is being delivered via a messenger (a human person, not the software) with a locked suitcase chained to his wrist.
I would really like to have my trust restored instead of completely changing the hard- and software architecture, which will take years.
Unfounded: Der Spiegel is a nice newspaper and I read it each morning. NSA spy my country, Swift (banks) and Belgacom (a telephone company). Through Belgacom they spied the UE parliament.
Any news about the mysterious MMcS service found on many SoHo routers amongst others some Cisco/Linksys branded devices?
https://communities.cisco.com/thread/11352
We are in the process of searching for two new routers/firewalls for our company. I had a device from the Cisco ASA 5500 seris in mind but after reading the documents and the “half-ass-way” Cisco is handling it (e.g. no firmware update for the 5500 to fix this) I seriously have doubts that Cisco wants to solve this problem.
It’s a real pitty.
Greetings from Europe
jb
Any news on the “newly” discovered ScMM vulnerability which can be found some SoHo routers?
Source: http://www.heise.de/newsticker/meldung/Mysterioese-Backdoor-in-diversen-Router-Modellen-2074394.html
Didn’t knew? This guy already pointed it out 2010 in your support forums: https://communities.cisco.com/thread/11352
If Cisco as well as the other affected US vendor didn’t knew about this hidden service one must assume that they are not documenting their software very well. Else it should be easily possible to explain what this service is intended for and why it was left open (on some models even over WAN)… Quality assurance anyone?
Hi Daniel,
Thank you very much for reading our blog and for taking the time to provide a comment. Please see the following Security Advisory for more information:
http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20140110-sbd
Thank you!
How the hell would NSA get their stuff onto the ASA in the first place ? According to their diagram it lives below the BIOS, wouldn’t such a low level require physical access to the ASA itself to add the “implant”? Given the leak states Jetplow “has been widely deployed” you would think Cisco labs could quickly find infected machines to verify if this is true. I reckon this is all fantasy and scare mongering, unless of course Cisco participated by adding the backdoor at manufacturing, and is now in denial. Who knows!
the corporate office of LM Ercisson has been bombarded with resuests (European customers) to sell comparable products currently being offered by Juniper and Cisco, there was panic among European customers related to the fear of NSA!
after looking at the list of the many, confirmed vulnerable devices it is quite unbelievable that the straight-up open port #32764 was never ever detected by the manufacturers.
how can you stand up for this?
While I understand the bind that Mr. Stewart has been put in by the NSA’s shockingly irresponsible and arrogant behavior, I’m afraid that I have to echo the comments made by many of the others who have left messages on this subject.
As a non-American IT Security professional, in view of the Snowden revelations (and remember… almost all of the information thus leaked, dates to the 2007-2009 time period… what’s the NSA up to NOW? I shudder to think…), I’m afraid to say that I can no more ethically recommend the purchase or use ANY American-made or -manufactured IT products, Cisco’s included, for my employers and customers.
I will be recommending that to the maximum extent possible, all American-made or -manufactured products should be removed from my employer’s / customers’ networks, and that a transition plan be drafted to phase out whatever other U.S.-made products remain, as quickly as is possible.
There is, unfortunately, NOTHING that you (Cisco) or any other U.S.-based IT supplier, can now do, to change my mind. The Snowden revelations — and even more importantly, the “stonewalling” U.S. government response to them — has proved beyond a reasonable doubt, that your government has no intention to reform its behavior; therefore, I have to assume that the U.S. government must, from now on, be considered as a “hostile attacker”, from a network security POV.
I must take counter-measures to ensure the confidentiality, availability and integrity, of my employer’s and customers’ valuable data, and the only halfway reliable counter-measure is to simply prohibit the use of your products and services. I sincerely regret this, as Cisco makes good products, but your government has, to put it crudely, “pi**ed in the well”, and to abuse another analogy, “Humpty Dumpty can’t be put back together again”.
Mr. Stewart — sir — this one is SERIOUS. Your company is really, really going to suffer because of the NSA’s abuses. I would suggest that you abandon any foreign sales targets that you may previously have had, and concentrate instead exclusively on the U.S. domestic market.
And thank Mssrs. Clapper and Alexander, while you’re doing it. I’m sure that you’ll get the same level of honesty from them, as did the U.S. Congress.
Sincerely
A Canadian IT Security Guy
Good thing, I know Russian language. Click to the below link:
http://www.cnews.ru/top/2014/01/10/raskryt_spisok_zhuchkov_anb_ssha_dlya_tehniki_cisco_huawei_i_juniper_foto_556040
Scroll down and it will take you to the scanned 50 pages NSA shopping list document for their agents. Those pages dated 06/24/2008, who knows what they have deployed on electronic devices, I am sure they are capable to shut down entire Internet by now and make devices completely dysfunctional. Shame on you Cisco allowing your devices to be vulnerable! You lost my trust!
Too bad, with all the secrets leaked by Snowden, there will be ZERO 0000000000000000 trust in any US firms.
I think many US tech firms have to leave the US and cut off all relations with US if they want to restore trust, something highly unlikely. Time to invest in Europe and Asia based firms. Bye bye to Cisco/Microsoft/IBM/Oracle/EMC/Dell/Juniper/Google/Apple. 🙁
Nice post and very creative too.
website design baton rouge
great post thank you so much
The potential client in then week’s steer describe around PalestinianPresident Mahmoud Abbas and even Israeli Major Minister Benjamin Netanyahu supplies noobvious environment designed for determination, or simply expectation. Abbas contains approved that talkswith an important unwillingness which will it seems to boundary at can’t stand, even while Netanyahu, his particular publicfirmly lurking behind your ex boyfriend, comes across as being modest burden to help make realistic credits. Core Eas
Sorry to say, but belief does not help.
Trust is gone and finding the truth will no be an option.
Therefore Cisco is out and for us – a german company with 600 employees – only security devices from german companies (bsi-certified) are of special interest any longer.
Think about speaking with your government, because spying at the whole world, doing device manipulation etc. and saying it’s for terror prevention is nothing we will believe and does not bring us back to buy USA-products.
I mentioned on December 31 in this comment thread that Cisco was extremely likely to have submitted a full copy of its router source code to NSA in order to get federal government permission to export those routers from the United States. John Stewart, I didn’t see any response to that. Did it happen, or didn’t it?
What has Cisco told NSA about how its routers work, that it doesn’t tell its ordinary customers?
In other words, what help did Cisco give NSA in subverting its routers — as required by the export controls?
John,
Thank you for following up and apologies for missing your earlier post.
I will respond more completely to your first post, but the brief answer is that Cisco has not helped the NSA (or any government agency from any country) to compromise our products. We are under no requirement to provide Cisco source code to the U.S. Government to secure export approval, nor do we.
Please see above for a more complete reply.
-jns
Hello John,
Its been a while since I’ve seen a new update regarding the checking tool. Is there any new information available now?
As a Cisco partner we (and our customers) really want to know more about the backdoor and how it got in the devices in the first place.
Further more when we can expect the checking tool and what is Cisco doing to prevent something like this happening again in the future?
Regards,
P. Bosman
Don’t be too hard on Cisco, they are obligated to keep quiet, can’t you see that? Please continue to buy their products so you can be safe from government spying. NSA does not spy on anyone except terrorists.
NSA has specific units of elite hackers, tasked with hacking into all kinds of hardware and networks. Further details on the subject reveal that these NSA hackers can hijack a Wi-Fi network, even while they are miles away. Keep posting this blogs .
nice article! I learned much from it!
____________
Do you want to earn money online?
Do you want to advertise your product online?
Clixsense is the best PTC site, it will give you all you want!
registration: http://www.clixsense.com/?6152434
We have no idea whether the NSA has “legally” compelled you, against your wishes, to do its bidding. We have no idea whether this response of yours has been compelled under threat of imprisonment. This is not paranoia — the other billion-dollar companies have provided precedent of gross deceit.
So let me summarize:
Three months later there is still no update to fix this NSA backdoors?
Most backlinking application is as powerful as No Hands SEO of promoting your entire website so long as you would like, you are able to set this backlink quality, just how many links you need each day and perhaps what anchorman texts you wish for any page of the entire website to get the finest ranks possible for your internet site.