Anyone played with Splunk?

I read Raffy's slides from his talk at HITB2007 on visualization, and this got me really interested log analysis through visualization. I flipped through his book (the first one, not the new Applied Security Visualization) at my local bookstore and liked it even more.

Around the same time, the company I work for finally decided that we need a log aggregation and analysis tool so that we know wtf is happening on our servers. So we called up Splunk, and I was impressed by a demo. Now we have them coming in to set Splunk up on our network and slurp up the logs from all our servers, firewalls, etc.

The demo impressed me. Hopefully this upcoming proof of concept doesn't disappoint.

Ohh, colored grep

that's not analysis

-id

I think you can get the same output from "cat|grep|awk|sort|uniq|sed|rrdtool" (maybe not in that order).. with a few parameters thrown in.. but then again, I'm not that good at rrdtool.. :)

EDIT: But yes, for PHB type, this is the best thing since double clicking!!!

--thrill

---

It is not the degrees you hold, but the mind you possess. - thrill



Edited 1 time(s). Last edit at 07/30/2008 11:56PM by thrill.

--Deleted by request--



Edited 3 time(s). Last edit at 07/01/2010 10:01AM by rsnake.

I am glad you liked the first security visualization book. I hope you will like the second one even better. It is much more hands-on. Would love to hear what you think when you had a chance to look at it. Drop me a note.

In any case, I feel compelled to chime in on the Splunk aspect of the discussion.

How much data are you dealing with? Just a few megabytes? Well, you might be okay with grep and awk. For me it doesn't cut it. And a lot of my customers have gigabytes of data to go through - quickly, securely, and centrally (check splunk.com for more details on all of that). Why do you use an RDBMS for your structured data? That's why I am using Splunk for my time-series data.

What I find myself doing a lot - aside from just looking at log files - is monitoring my systems for CPU and memory utilization. In addition, I like to know what processes are running and what ports are open at any point in time. I just pipe netstat and ps into Splunk and have instant access to that information - for any point in time. I can then _interactively_ report on the open ports - I can see when they disappeared and when new ones showed up. I actually have an alert set up so that I get notified anytime there is a new port showing up. And in case something every happens to my boxen, I have a pretty nice audit trail. That's just one thing I like to do in addition to monitoring my logs. I could give you dozens of more use-cases, but in the end, it's up to your imagination what you want to do with Splunk.

I encourage you to download Splunk (it's free!) and put it on your laptop. If you still think you are better off with grep and awk, you should then enable the file system monitor and the registry monitor (on Windows). I am sure you will like that.

Anyways. I will stop now. Check http://www.splunkbase.com to see some applications that people built around Splunk.

-raffy from splunk>

PS: You can use the search language (cluster, correlate, rare, similar, etc.) to do "real" analysis with Splunk...

--
raffy @ secviz.org -- http://secviz.org

@raffy - Looking at your website it's easy to recognize your love for graphs, and while I agree that some things make much more sense in a graph, there are specific instances where the raw data, even when graphed, might not amount to any substantial information.

Take this as an example:

Long ago I worked for a company that did network security. My job was pretty much to look at logs and try to identify attack patterns, then I would create an ACL and deploy that onto border routers. The data in bulk when graphed would show a specific pattern of attacks, be it NetBus, BackOrifice, or just plain old IP/port discovery scans. It was great for all of that. However, hidden within these logs were very specific, targeted attacks that were missed by the graphing, and even when going back a full 2 weeks worth of data, the graph never showed a single IP address that was scanning one port every 12 hours.

It wasn't until I remembered the source IP that allowed me to start putting 2+2 together, and while grep/awk/sed/uniq, etc., etc. may not be the most glamorous tools, it's the fact that I am not bound by someone else's logic and limitations that is the key for me, and many others like me.

I'm almost positive that I could dump those old logs (if I still had them) into any of a dozen log monitoring utilities and they all probably would miss the little attack.

But yes, they would provide some nice pictures! ;)

--thrill

---

It is not the degrees you hold, but the mind you possess. - thrill

thinking of splunk as a simple grep is such an understatement of what it does.

some motivating questions:

1. how do you grep for may 5th in a logfile? is it 5/5/08? may 5? 5 may? is it the number of seconds since jan 1, 1970? btw, What timezone is that logfile in? What if you want only the events within the nearest hour? across all of your devices and all the log formats?

2. splunk doesn't hide the raw logs from you. they're an integral part of the search results. so if there are specific, targeted attacks that you're talking about, you'll find them a hell of a lot faster with splunk than you would grepping or walking the files manually.

3. also, if you've got a crapload of logfiles, like most of us do, at least you get the benefit of not having to walk the entirety of the files every time you iteratively refine your grep.

4. the query language is able to do things that are clunky in grep at best. e.g., requests like "show me all loglines that occur between 5/5/08 10am PST and 5/5/08 11am PST that referred to an IP address in this range and came from one of our non-cisco routers" is a trivial idea to express in splunk. even if i had tens of gigs of logfiles i'd expect to be able to get an answer to that kind of question in a small number of seconds from splunk, including answers to each of the various refinements of the query as i'm going in and learning more. e.g., you start with "something went wrong at 10am", to "something went wrong at 10am and seems to be related to this set of boxes", to "it seems to be something with routing", etc.

if you spend a lot of time doing this kind of work (which obviously you do) and the splunk demo you got didn't impress, you might want to get someone else to show you what's up.

also, that's just the search/grep-ish part of the thing, which is the simplest part of the offering.

andre

p.s. full disclosure: i used to work at splunk. i still think they have both a great company and a great product. fun and gets shit done.

You really, really, really don't understand what splunk is. Nothing in splunk is hardcoded. It's not a bunch of pretty graphs that are hardcoded and will miss what you want.

You can see the search results (text) or graphs of *whatever* you want.
Splunk has an exceedingly powerful search language that does everything you want, usually subsecond. Here's some searches that relate to what you said...

show me the top src_ip addresses:

$ splunk search "sourcetype=syslog | top src_ip"

search all my syslog files, group all events that have the same src_ip into a transaction, and finally just show me for each src_ip how many times it accessed each port, sorted from the most hits to the least...

$ splunk search "sourcetype=syslog | transaction field=src_ip | sort port,-count | fields count,src_ip,port"

I could go on forever, making searches that find anomalous values, extract out keyterms, find complex transactions, etc., none of which are hardcoded!

again, none of this is hardcoded, it's damn powerful, flexible, subsecond, handles gigabytes with no sweat, is free, and make pretty graphs to boot.


thrill Wrote:
>
> My job was pretty much to look at logs
> and try to identify attack patterns,
>...the graph never showed a single IP address
> that was scanning one port every 12 hours.
> ...
> while grep/awk/sed/uniq, etc., etc. may not be the
> most glamorous tools, it's the fact that I am not
> bound by someone else's logic and limitations that
> is the key for me, and many others like me.
>
> I'm almost positive that I could dump those old
> logs (if I still had them) into any of a dozen log
> monitoring utilities and they all probably would
> miss the little attack.
>
> But yes, they would provide some nice pictures! ;)

somehow I'm flashing back to ipcops...

Quote

how do you grep for may 5th in a logfile?

Umm.. depends on the log file?

Quote

You really, really, really don't understand what splunk is.

Maybe so, but as requested by the original poster, I am expressing my OPINION. Maybe one day when my spam volumes drop and I want to sign up for more spam, I'll fill out your questionnaire and download it, but until that day comes, my opinion on the software with the funky (or is that splunky?) ass name will have to suffice.

--thrill

---

It is not the degrees you hold, but the mind you possess. - thrill

I totally understand and still appreciate the power of grep, sed, awk, etc. Still use them daily for many tasks.

However,I can also remember the days before the internet and having to log into dozens of machines and using ftp to get the information needed. Or using ftp mode in emacs to make browsing for porn easier ;)

*sigh*....those were the days...

But then came Mozilla with the browser and internet search engines and life got easier. Now I can search across hundreds of machines and thousands of files without knowing anything but a few terms of what I'm looking for. To me, this same analogy applies with Splunk. Except instead of typing 'naughty college girls', I'm typing things like 'failed password host="10.2.*' and getting an equal amount of joy.

Ok, that last part was a lie, but in a geeky sort of way it is pretty cool :) Check it out...

thrill Wrote:
>
>> how do you grep for may 5th in a logfile?
>
> Umm.. depends on the log file?

so you have to custom tailor your grep command for each file? that's silly. especially if you are trying to find problems across multiple files. It's also close to impossible to get lines between 8-11am without listing them all or using regex. What about normalizing timestamps across timezones? This is as trivial in splunk as a click on a calendar or in the search language.


> Maybe [I am uninformed about splunk], but as requested by the original poster,
> I am expressing my OPINION.

but your opinion is less than worthless if you've never tried something. it's misleading. should I post to the toyota forum that toyota's are uncomfortable without ever sitting in one? should I say that they are the least safe cars without any proof, and then defend it by saying that people only asked for my OPINION?

> Maybe one day when my spam volumes drop and I want to sign up for more spam,

It's lame to have to enter an email address, but understandable. Jez, you're getting a free product. But your excuse is doubly a lame and you know it.
Make a new gmail account. It takes 30 seconds. Then never use it again.
Ta-da!

@IPEuropean - AKA Official Rhymes with Junk Rep.

Quote

but your opinion is less than worthless if you've never tried something.

Well, my opinion was formed from the information I gathered on your website, which in case you hadn't noticed, is the 'selling points' of this free product, and the reason why I should even waste 30 seconds of my day evaluating something which I really do not have a need for. I do not have logs with differing time zones, if I did, maybe the logical thing would be to set all my machines up for UTC, but I guess that solution is just a little too advanced for you great minds.

2nd, this particular website is actually not an official "Splunk Product Review", if it were, then you would have all the right to attack my views and opinion, but since it is not, I suggest you take a step back and realize that you have absolutely no business attacking me or anyone else on this site for our opinions.

As for the grep commands, it is really not that hard to type ^regex1^regex2 if I wish to look for something different.

In short, you have your tools and probably your religion as well. I do not share your tools nor your religious views. I am not trying to cram my tools and religious views down your throat, so I'd appreciate the same from you.

@strayhud

Quote

Ok, that last part was a lie, but in a geeky sort of way it is pretty cool

Heh.. glad you put that there because I was starting to think there was something seriously wrong with you.. and yes, the good old days of archie and veronica piping into grep and spawning multiple ftp connections.. ahh the memories.. ;)

--thrill

---

It is not the degrees you hold, but the mind you possess. - thrill