Graylog WebHDFS Output Plugin
An output plugin to write messages to Hadoop HDFS over WebHDFS.
Tested with Hadoop 2.7.0.
The WebHDFS implementation uses modified code from webhdfs-java-client.
Getting started
To start using this plugin place this jar in the plugins directory and restart graylog server.
Following parameters can be configured while launching the plugin
- Host - IP Address or hostname of the HDFS name node
- Port - WebHDFS port configured in HDFS.
- Username - Username of pseudo authentication (currently kerberos is not supported)
- File path - Path of file to store the messages. File name can be formatted with message fields or date formats. E.g ${source}%Y%m_%d.log for storing the messages based source and day.
- Message Format - Format of message to be written. Can be formatted with message fields like ${timestamp} | ${source} | ${short_message}
- Flush interval - Interval in seconds to flush the data to HDFS. Value of 0 means immediate.
sivasamyk
free!
We have recently created a stream with this ouput, but we it shows following error message:
graylog_1 | 2016-09-29 08:57:30,394 INFO : org.graylog.outputs.hdfs.WebHDFSOutput - WebHDFSOutput launching...
graylog_1 | 2016-09-29 08:57:30,394 ERROR: org.graylog2.outputs.OutputRegistry - Unable to fetch output 57ebedcc71846f0001c6d857, fault #5
graylog_1 | com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Non-positive period.
graylog_1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2207) ~[graylog.jar:?]
graylog_1 | at com.google.common.cache.LocalCache.get(LocalCache.java:3953) ~[graylog.jar:?]
graylog_1 | at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4790) ~[graylog.jar:?]
graylog_1 | at org.graylog2.outputs.OutputRegistry.getOutputForIdAndStream(OutputRegistry.java:102) [graylog.jar:?]
graylog_1 | at org.graylog2.outputs.OutputRouter.getMessageOutputsForStream(OutputRouter.java:44) [graylog.jar:?]
graylog_1 | at org.graylog2.outputs.OutputRouter.getStreamOutputsForMessage(OutputRouter.java:64) [graylog.jar:?]
graylog_1 | at org.graylog2.buffers.processors.OutputBufferProcessor.onEvent(OutputBufferProcessor.java:132) [graylog.jar:?]
graylog_1 | at org.graylog2.buffers.processors.OutputBufferProcessor.onEvent(OutputBufferProcessor.java:51) [graylog.jar:?]
graylog_1 | at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
graylog_1 | at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
graylog_1 | at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
graylog_1 | Caused by: java.lang.IllegalArgumentException: Non-positive period.
graylog_1 | at java.util.Timer.schedule(Timer.java:247) ~[?:1.8.0_102]
graylog_1 | at org.graylog.outputs.hdfs.WebHDFSOutput.(WebHDFSOutput.java:80) ~[?:?]
graylog_1 | at org.graylog.outputs.hdfs.WebHDFSOutput$$FastClassByGuice$$4089382c.newInstance() ~[?:?]
graylog_1 | at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?]
graylog_1 | at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[graylog.jar:?]
graylog_1 | at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?]
graylog_1 | at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?]
graylog_1 | at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[graylog.jar:?]
graylog_1 | at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[graylog.jar:?]
graylog_1 | at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[graylog.jar:?]
graylog_1 | at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:776) ~[graylog.jar:?]
graylog_1 | at com.sun.proxy.$Proxy101.create(Unknown Source) ~[?:?]
graylog_1 | at org.graylog2.outputs.MessageOutputFactory.fromStreamOutput(MessageOutputFactory.java:51) ~[graylog.jar:?]
graylog_1 | at org.graylog2.outputs.OutputRegistry.launchOutput(OutputRegistry.java:143) ~[graylog.jar:?]
graylog_1 | at org.graylog2.outputs.OutputRegistry$2.call(OutputRegistry.java:137) ~[graylog.jar:?]
graylog_1 | at org.graylog2.outputs.OutputRegistry$2.call(OutputRegistry.java:133) ~[graylog.jar:?]
graylog_1 | at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4793) ~[graylog.jar:?]
graylog_1 | at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) ~[graylog.jar:?]
graylog_1 | at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) ~[graylog.jar:?]
graylog_1 | at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) ~[graylog.jar:?]
graylog_1 | at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[graylog.jar:?]
graylog_1 | ... 10 more
graylog_1 | 2016-09-29 08:57:30,398 ERROR: org.graylog2.outputs.OutputRegistry - Output 57ebedcc71846f0001c6d857 has crossed threshold of 5 faults in 30 seconds. Disabling for 30 seconds.