Security Logging in an Enterprise, Part 1 of 2

Logging is probably both one of the most useful and least used of all security forensic capabilities. In large enterprises many security teams rely on their IT counterparts to do the logging and then turn to the IT logging infra when they need log information. That in itself isn’t bad; however, the needs/requirements for IT may not be a 100% fit for a CIRT. Read on to find out how we handled it.

While IT maybe more interested in real-time feedback on, say, a disk going bad, the CIRT may be less interested in that and more interested in who logged in to a server 6 months ago. A CIRT may in fact turn off capacity planning and hardware informational logging to facilitate longer-term storage of logs that are interesting from a security perspective. Similarly IT may want to bulk repetitive queries for hardware failures that could slow the CIRT’s abilities for investigative searches. These diverging requirements, along with resource contention, make it worthwhile to consider a separate high-integrity data store for sole use of a CIRT. At Cisco the CSIRT started logging data itself before IT’s requirements for integrity were as stringent as they are today. Many years back IT was happy with keeping system logs, mostly on the systems themselves. So we set up a syslog reflector capability and kept a small subset of logs off system. One of the big early cases for us was solved when we compared the logs on system vs. our own off-system copy. The hacker (masquerading as several users) neatly removed everything he did on the system, so by examining every extra log file on the off-system storage we had a complete list of everything the hacker did. Our logging needs and capabilities soon outstripped IT’s and now Cisco IT has modeled their logging infra after our setup.

A couple of points to consider: We don’t do logging for the company. This was an initially unpopular decision due to a certain duplicity of efforts. My perspective on this was that I want the CIRT to spend more time on examining and responding to incidents from the data than we spend on organizing and administering the data. I knew if we became the defacto logging infra for the company we would be standing up a service that would come with considerable uptime and response considerations that would pull as away from our core – responding to incidents. Combine that with a golden rule for CIRTs that we should have no conflicts of interest in examining issues with IT infra. If we are also providing that same infra, we are no longer an external investigative organization and thus lose our neutrality. It was a long battle, but one that ended up with IT owning and making a better logging solution long term. We still run our own logging infra; however, we don’t own things like company logging for compliance so we have flexibility to pick and choose the things we log (or drop). We have detailed logging standards that we enforce as well as documenting what to log and how long to keep the logs. It starts with the following:

The owner of the system or application that generates logs is responsible for meeting all regulatory compliance requirements relevant to logging for the system or application in question. This includes ensuring that logs are archived and retained for the time period mandated by the regulatory requirement (also see the ‘Records Management’ link below for help in determining log retention requirements). Though the Cisco CSIRT may elect to receive and store a copy of the logs (or some security-relevant subset of the logs) the CSIRT will not assume responsibility for the regulatory requirements of the system or application in question. If requested by the Cisco CSIRT, security-relevant logs should be forwarded in real-time to a centralized log collection server as specified by the Cisco CSIRT. In cases where the system or application cannot be configured to forward logs the CSIRT may request authentication credentials for the device to allow for periodic “pulls” of log data.

So what do we log? See an upcoming part 2 for that!

Share