We all have that one incident that made us totally...

Posting as :

This was really recent in a 10 year career and the incident itself wasn’t bad—but more by luck as it could have been bad. The change in approach is more because of why I missed it + some reporting work I was doing at the same time.

We got notified by a third party, no name but if you list off the orgs, agencies, or that a threat actor was on one of our servers. It doesn’t help that they were kind of vague other than giving me one IP.

It turned out to be one of the hosts in our VDI environment. I did have some logs; session from the authentication to the vdi environment and network firewall, but the parsing of those logs wasn’t great and this was due to CVE-2023-4966 (session hijack on netscaler, and MFA was bypassed due to this) which itself did not leave any real evidence — except that the source and client IP would differ. Since that is a rare case for legitimate use, both fields weren’t captured outside of the original message text. There were no logs collected on the host and because it was VDI, the VM would periodically get destroyed and I could not go back to grab this information. Had they gotten a privileged account it could have been much worse.

Now for the reporting, trying to rework how I was doing our vulnerability reporting had already gotten me to the point of reevaluating how I’m handling that data and its processing. With the timing, I saw the same difficulty with logging.

The lesson learned here:

- I cannot possibly be the only one responsible for understanding the logging functionality, format, and options of every single thing in our environment. The system owners need to step up to their role here.
- Getting this to happen also means our logs can’t go into a black hole that they’ll never see after making the configuration changes. They should be able to see the logs and identify when something’s off. If logging is already happening then maybe this should also include more data than strictly security (metrics, applications logs, etc— all of which also can be useful for security )
- We have a real need to standardize how we’re documenting our environment, configuration, and services. This is also basic but previous leadership actively worked against me trying to get it done. So I’d be primarily focused on ensuring that detecting and responding to threats based on what we did have could be done as quickly as possible.

The approach I’ve been taking for the last half year has been based on this.

My top priority is transitioning our security program to a collaborative model where those system owners are responsible for understanding their system’s security and monitoring for security notifications. Security will be responsible for maintaining, monitoring, and auditing the inventory, configurations, and data being collected. Identified outside of defined change management need to be validated and documented as a change (as the change management itself is that validation and can be accepted as is). Insecure configuration, vulnerabilities, and other findings are included and sent to those owners. Data collected should be provided to those folks, as programmatically as possible so queries and automations can be made without having to understand the minutia of the whole environment. Networking or Server people can make changes targeting what they know to be wholly accurate and more importantly where to find it (one webui/api vs 10 different manners of documenting it) or simply sync the “true” inventory to what they’re currently doing. I won’t need to worry about constantly having to redo discovery to catch changes either not documented or spread documentation across network shares, wikis, and spreadsheets. I can focus on the overall security instead of constant battles with searching for info or trying to suss out some log means.

But it all really comes down to ensuring we have the information to do that, it’s continuously up to date, and it’s relevant and usable to the people I need securing their stuff.

Folks just starting in security, every ciso will say “you can’t protect what you don’t know” to the point it’s like their Pokémon word but lemme tell ya. It goes well beyond just knowing. You can’t secure what you can’t validate and you can’t validate or secure what you don’t understand. “Information silos” and knowledge gaps in your environment are as big of a threat as the actual threat actors. You’ll miss attacks, of catch them much later. You’ll declare an incident—or breach—when it may not be necessary. You’ll spend hours searching for info that you shouldn’t need to.

Posting as :

Download the Fishbowl app to
unlock all discussions on Fishbowl.