Dedupe Processor

Remove duplicate log records across a set of events using Match or Ignore comparison on specified fields to reduce log chatter.

Description

The Dedupe processor removes duplicate values from log data.

Use

This processor is most useful for reducing “chatter” in logs. The overlap of data across fields is the key to having this processor work effectively. This processor will emit the first matching record of the set of records that are being compared.

Configuration

There are three options to configure for this processor.

Option	Description	Example
Number of Events	The number of events to compare across. Limited to 5000.	5000
Comparison Type	Match will remove duplicate records based on the specified fields. Ignore will remove duplicate records based on all fields except those specified.	Match
Fields	The field or fields to apply the Dedupe processing to.	.foo

Example - Match

Before

{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, 'bat': true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "qux", "baz": 5}

Dedupe Options

Option	Value
Number of Events	5
Comparison Type	Match
Fields	.foo

After

{"foo": "bar", "baz": 1}
{"foo": "qux", "baz": 3}

Example - Ignore

Before

JSON
{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "qux", "baz": 4}
{"foo": "corge", "baz": 5}

Dedupe Options

Option	Value
Number of Events	5
Comparison Type	Ignore
Fields	.baz

After

{"foo": "bar", "baz": 1}
{"foo": "bar", "baz": 2, "bat": true}
{"foo": "qux", "baz": 3}
{"foo": "corge", "baz": 5}