ZAT may ignore contents after character '#'

[kascas]

Hello, ZAT helps me a lot, but I think ZAT fails to handle '#' properly. In a sql injection event, the url contains character '#' and ZAT ignore the contents after it. The two picture can describe this:

This picture is Zeek's http.log, you can see the url is /dv/vulnerabilities/sqli/?id=1'+and+1=1#&Submit=Submit

But when I debug my python program, the uri only record url as /dv/vulnerabilities/sqli/?id=1'+and+1=1 and all fields after uri is None. That is ,missing all contents after '#'.

My usage of ZAT is like this:logdict = json.loads(LogToDataFrame().create_dataframe(fullname).to_json(orient='records'))

[brifordwylie]

Hmm... okay.. thanks for the problem description... I'll try to take a look at this...

[brifordwylie]

Okay.. so this turns out to be more difficult than expected. The reason is that we're utilizing the Pandas comment parameter to clip out both the Zeek header fields and the closing '#close' tag that zeek puts on the end of a log. Anyway, still looking at optional/approaches here...

[kascas]

So is it possible to use regular expression or other methods to make a distinction between header fields and values?

[brifordwylie]

Certainly, but the higher level issue is do you let Pandas take care of the details or do you 'roll your own'.. and rolling your own is always a bigger effort. So I'll do some experimentation in the next few days and come up with something.

By the way.. I noticed that you aren't even using the DataFrame (as least in your use case/snippet)

logdict = json.loads(LogToDataFrame().create_dataframe(fullname).to_json(orient='records'))

# Minutia: If you're going to use this call... there's no need to write json and then read back json...
logdict = LogToDataFrame().create_dataframe(fullname).to_dict("records")

So if you just want a list of dictionaries you can just use this code (which works with comments in url)

from zat import zeek_log_reader

reader = zeek_log_reader.ZeekLogReader("http_with_comment.log")
logdict = [row for row in reader.readrows()]

print(logdict)

[{'ts': datetime.datetime(2013, 9, 15, 17, 44, 27, 668082),
  'uid': 'CyIaMO7IheOh38Zsi',
  'id.orig_h': '192.168.33.10',
  'id.orig_p': 1031,
  'id.resp_h': '54.245.228.191',
  'id.resp_p': 80,
  'trans_depth': 1,
  'method': 'GET',
  'host': 'guyspy.com',
  'uri': '/home/blah/example#comment=1+1',
  'referrer': '-',
  ...
 {'ts': datetime.datetime(2013, 9, 15, 17, 44, 27, 731702),
  'uid': 'CoyZrY2g74UvMMgp4a',
  'id.orig_h': '192.168.33.10',
  'id.orig_p': 1032,
  'id.resp_h': '54.245.228.191',
  'id.resp_p': 80,
  'trans_depth': 1,
  'method': 'GET',
  'host': 'www.guyspy.com',
  'uri': '/home/blah/example#comment=2+2',
...

[kascas]

Thanks for your solution, and ZeekLogReader do help me get a correct dictionary from logs. Perhaps I need to read document (or source code) more carefully... So, thank you again for taking time to solve my problem : )

[brifordwylie]

@kascas no problem. Thanks for reporting this.. I'm going to reopen this ticket.. for folks using the Pandas Dataframes this will need to be fixed. :)

[Bakloul1998]

Hello Mister @brifordwylie , can you help me on how to use and run the code ?

[brifordwylie]

@Bakloul1998 that's a pretty broad question. ZAT can be used in a bunch of different ways.. the documentation contains many code examples and notebooks: https://supercowpowers.github.io/zat/ please start with the examples page https://supercowpowers.github.io/zat/examples.html.