How does a SIEM project usually go?
You purchase a license (through an RFP process or not), the integrator comes, gathers information about your environment, two weeks later they come to set up the configuration and then you start seeing beautifully ingested logs from all across your environment, allowing you to define meaningful correlation rules.
Well, of course, that’s nonsense. It’s never as smooth and straightforward, no matter what the vendor claimed in their datasheet or proposal. There are all sorts of complications:
- Missing project plan – “we’ll just get everything done” is not a project plan. It should be well structured, starting with the information gathering, configuration preparation, installation (even with cloud options there may still be an on-prem collector/forwarder)
- Slow in obtaining the right permissions – if the integrator doesn’t communicate clearly what permissions are needed for each source, the credentials might require some chasing. A meeting about “where is the AD event log service account” should not happen, but it doesn’t.
- Missing support for your log sources – yes, every vendor claims they can consume any source, but organizations learn the hard way that it’s rarely true. Network appliance vendors are way too sloppy to follow a set of standards consistently, so “syslog” and “CEF” may often mean nothing in the wild, not to mention undocumented (or improperly documented) vendor-specific formats and “extensions”. A series of “hacks” are the better outcome in these situations (e.g. preprocessing the logs with bash scripts in order to fit them into the known formats) and leaving the source out entirely is what typically happens.
- Lack of cloud integration – AWS, GCP, Azure, Microsoft365, Google Workspace, Salesforce, Dropbox, Okta – your company is likely using at least some of those, and it’s crucial not to miss any of them in your SIEM. Legacy SIEMs don’t have cloud (IaaS/SaaS) support, but most vendors nowadays do have some integrations. But without your cloud, your SIEM is pointless.
- Not extracting enough structured information for meaningful correlation rules – having basic info from the logs is one thing, but in order to implement all the required correlation rules, you need more. Regex, if supported, is a good way to parse things, but it may not be sufficient. JSONPath, XPath, CSV/TSV and other extraction approaches must be available in either the SIEM, or the collectors/forwarders, or both.
- Forcing an agent – “agentless” collection sounds great, but an integrator may insist that their agent is “lightweight, easy to manage and secure”. Pushing a binary to all of your endpoints in a streamlined process is always scary because we can never fully trust any binary (remember Sunburst). But if there are not enough options available in the collector/forwarder, an agent might be the only option to collect data and that complicates things, especially if you already have AV/EPP/EDR.
The collection is the hardest part of a SIEM project – if it goes well, everything else follows nicely. But too often it doesn’t. And that’s because either the SIEM vendor has oversold its log collection capabilities, or because the selected partner is not proficient enough to be able to set it up. But, needless to say, it’s never your environment’s fault.
And skipping the hardest sources (which unfortunately happens way too often) is one of the first signs of a failed SIEM project. After all, it should aggregate everything in order to let you derive value.
A SIEM project can fail in other ways as well – through exploding costs, or by being too complicated for a team without expertise in the particular product, by generating too many false positives.
How to Prevent a SIEM Project from Failing?
One important step to mitigate a lot of risks of a failed SIEM project is to have a POC (doesn’t matter if it’s free or paid, though the paid one might lead to a “sunk cost fallacy”). You can’t really do a one-step RFP and be sure that things will work out. This is more complicated in the public sector, but even then you can have a two-step project with milestones and KPIs, even though it may require extra procurement wizardry.
Some vendors allow POCs but don’t offer assistance through their partners. In other words, the POC is not part of their partnership program and so partners don’t know exactly how to approach that. So you are left on your own with an OVA to unpack and documentation to dig into.
So, to summarize the ways to avoid failed SIEM projects:
- Have a POC and try to connect as many sources as you can during the POC. Seek support from a vendor partner or from the vendor directly
- Make sure your entire environment, including cloud sources, can be connected and properly parsed. Verify that you can plug in custom parsers easily, not just regex, but other expression languages and even custom code
- Let the entire team play with the tool and confirm that it’s easy to use
- Choose a pricing model that won’t kill your budget after you tweak a few log configurations. “Land and expand” is a great business model for vendors, but not as great for SIEM customers
- Have a project plan, and insist on one from the integrator
Bozhidar Bozhanov is a senior software engineer and solution architect with 15 years of experience in the software industry. Bozhidar has been a speaker at numerous conferences and is among the popular bloggers and influencers in the technical field. He’s also a former government advisor on e-government, transparency, and information security.
