Back to listing

Windows DNS

Content Pack

A Windows DNS content pack for graylog.

reighnman
free!

Published

23 Sep 10:53

Last Push

11 May 09:36

Marketplace Rating

Discussion

24 Comments

Your Rating

Please sign in to rate this add-on.

Comments

rka 7 months ago

Hello I have the old ThreadID problem with graylog version 2.1.1. I tried the old fix, and creating a new index template, but not working.
Details here: https://github.com/reighnman/Graylog_Content_Pack_WinDNS/issues/6

Any ideas?

JulioQc 9 months ago

Nvm my comment about specifying GROK field datatype to string. It doesn't work on the long run.
Custom mapping as suggested works well.

JulioQc 10 months ago

Yes but forcing datatype with grok is simpler then making custom index mapping (imho), followed by a cycling. I would suggest updating the content pack with this minor change or at the minimum recommend implementing custom mappings to avoid the issue.

reighnman 10 months ago

@skear cycling the deflector is only a temporary fix - you must create a dynamic template on your ES cluster that forces ThreadID to be a string field. If you cycle the deflector again and the first ThreadID entry that gets inserted begins with an INT, it will break it again until you cycle again and get an alphanumeric entry first

JulioQc 10 months ago

Possible bug (and workaround suggested) with the GROK pattern: https://github.com/Graylog2/graylog2-server/issues/2645

JulioQc 10 months ago

Kudos to SandMouse for his instructions to correct the capture (way simpler then changing NXlog version). One can also modify the JSON to make the change.
Also, consider changing 'dns' in the JSON in for the proper Collector Input ID (ie 578e3f830ae2f10b113845fe) for the Dashboard to work correctly, if you are using Collectors that is.

skear 11 months ago

I was able to get this working with Graylog 2.0 but I ended up having to use an older version of nxlog (2.8.1248). Nxlog 2.9.1504 doesn't send the full_message field which creates a problem. I tried using shortmessagelength -1 in the of the nxlog config file but it was still truncating the message field to 64 characters.

After I started using nxlog 2.8 I was receiving messages with the full_message field but I was seeing errors about failed to parse ThreadID in the graylog log files. This issue was fixed by going to system \ indicies \ maintenance \ manually cycle deflector. After doing that the errors went away and the messages were being parsed correctly.

To clarify , since I used nxlog 2.8 I did not need to make any modifications to the content pack json file.

Now that it's finally working I'm really happy with the data this content pack is producing!

SandMouse about 1 year ago

I used the following patch to change the fieldname in the json file:

--- content_pack.json 2016-05-11 10:49:18.000000000 +0200
+++ content_pack_chaged.json 2016-05-11 10:41:02.000000000 +0200
@@ -22,8 +22,8 @@
"converters" : [ ],
"order" : 0,
"cursor_strategy" : "COPY",
- "target_field" : "full_message",
- "source_field" : "full_message",
+ "target_field" : "message",
+ "source_field" : "message",
"condition_type" : "REGEX",
"condition_value" : " PACKET\\s+[A-Z0-9]{16}\\s+UDP|TCP\\s"
}, {

SandMouse about 1 year ago

@eduham It seems as if I've ran into the same problem the one you are describing. The query "SourceModuleName:dns AND Context:PACKET" won't return any results while "SourceModuleName:dns" does return plenty. Were you able to solve this? Anyone else who might me able to help?

eduham about 1 year ago

ok, i´ve seen my problem is anotherone. my shown message is right but i can´t get a full_message. In the extractor testview i´ve get this message: Could not load an example of field 'full_message'.
Anyone an idea?

eduham about 1 year ago

Thx 4 this great Content Pack. Logging works on my system but the dashboard wouldn´t show something. Now i´ve read the searchquery and see that the script is searching for "SourceModuleName" and "Context". My problem is that the context field is unknown. Is there a problem with the grok_pattern? +%{NOTSPACE:Context}
Here my Pattern for "NOTSPACE"
"name" : "NOTSPACE",
"pattern" : "\\S+"

wjanowiak about 1 year ago

See comments here - was able to get it working with some massaging. Temporary fix for now.

https://github.com/reighnman/Graylog_Content_Pack_WinDNS/issues/5

mbuyukkarakas over 1 year ago

Sorry my friend.
I tried with my GL 1.3 but its not working. I tried Pajje99's solution but its not working either. Also Pajje99's pattern is not completely pasted. It gives warning when I try to edit the GROK pattern.

Pajje99 over 1 year ago

I got this tip from another user and it works now.

Try updating WINDNS_TIME to:

((?:0?[1-9]|1[0-2])[\/\.-](?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])[\/\.-](?>\d\d){1,2}\s(?!\d\d){1,2}\s(?!\d\d){1,2}[\/\.-](?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])[\/\.-](?:0?[1-9]|1[0-2])\s(?!

reighnman over 1 year ago

Have you tried opening an issue on github?

Kopke over 1 year ago

Pajje99: Same problem here. Using Windows 2012R2 server, latest version Graylog appliance & nxlog-ce-2.9.1504

scub4st3v3 over 1 year ago

Pajje99: Were you able to solve this? Having the same problem.

Pajje99 over 1 year ago

HI!

I can not get this contenpack to work right. It seems that the grok pattern is not working right because it will not extract right.

I get:
full_message
2015-12-02 21:32:41 15F8 PACKET 0000009946EF00A0 UDP Snd 8.8.8.8 ae34 Q [0001 D NOERROR] PTR (3)142(3)146(3)248(2)17(7)in-addr(4)arpa(0)

and

message
2015-12-02 21:32:41 15F8 PACKET 0000009946EF00A0 UDP Snd 8.8.8.

and nothing is showing on the dashboard.

reighnman over 1 year ago

123dev: I have not experienced that issue using 2008 R2 DNS servers with the NXLog example in the doc so I'm not sure

123dev over 1 year ago

This is a great ContentPack, thanks for sharing.

One question, how do you mange to keep the dns.log file not disappear on you (on rotation).
The thread [http://nxlog.org/question/603/windows-dns-log-20082012] discusses the issue

I wonder if anyone has managed to figure out a way around this limitation.
Thanks

reighnman over 1 year ago

Woops, I forgot about this.. I had to create a mapping template in ES to hard set the ThreadID field type as String (which I'll add to the doc) but for reference something like: (to create template called graylog that matches all indexes starting with "graylog")

curl -XPUT localhost:9200/_template/graylog -d '
{
"template":"graylog*",
"settings":{
"index.refresh_interval":"30s"
},
"mappings":{
"message":{
"properties":{
"ThreadID":{
"index":"not_analyzed",
"type":"String"
}
}
}
}
}'

Basically with ES when a new field is created it dynamically assign it's type based on the first insert, which if the first value you're inserting is all integers it assigns the field as int (which ThreadID is 90% of the time). ThreadID can also contain alphanumeric, so when an alphanumeric ThreadID comes around later it can't insert it into the field because ES is requiring an int value.

Sorry!

richardd84 over 1 year ago

Yep, this is what it was the field "ThreadID" was in use already as a numeric type, i ended up changing the field name within the grok pattern to %{WINDNS_THREADID:WinDNSThreadID} which resolved the issue.

Thanks.

lennartkoopmann over 1 year ago

The field "ThreadID" seems to be mapped as a numerical type. Do you have that field name in some other messages where it is a number and not a string? Try do a manual index cycling to reset the mapping (System -> Indices -> "Maintenance" dropdown) and try renaming the field this content pack creates if that does not work.

richardd84 over 1 year ago

Seem to have a problem with the grok pattern for the ThreadID in the DNS debug logging.

2015-10-26_20:39:33.72886 [185]: index [graylog_1], type [message], id [a93a3f83-7c21-11e5-a1dc-005056846a42], message [MapperParsingException[failed to parse [ThreadID]]; nested: NumberFormatException[For input string: "0AD4"]; ]
2015-10-26_20:39:33.72886 [186]: index [graylog_1], type [message], id [a93a3f85-7c21-11e5-a1dc-005056846a42], message [MapperParsingException[failed to parse [ThreadID]]; nested: NumberFormatException[For input string: "0AD4"]; ]
2015-10-26_20:39:33.72886 [187]: index [graylog_1], type [message], id [a93a3f87-7c21-11e5-a1dc-005056846a42], message [MapperParsingException[failed to parse [ThreadID]]; nested: NumberFormatException[For input string: "0AD4"]; ]

Any ideas?

Please sign in to comment.

Back to listing