Skip to content

Commit 42a5cfa

Browse files
author
anders-wartoft
committed
1.1-2 StringInputItem, code cleanup, documentation fixes and GuardFilter fix
1 parent 259ae63 commit 42a5cfa

26 files changed

Lines changed: 540 additions & 105 deletions

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
target
2+
3+
.vscode

‎README.md‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ java -jar target/LogGenerator{version}.jar -i kafka -ci test3 -t OUTPUT -b 192.1
1818
When running the last command, press Ctrl-C to see the gaps in the received data. Since we started the counter on 100, there should at least be one gap: 1-99.
1919

2020
### Latest Release Notes
21+
#### 1.1-2
22+
Update of documentation. E.g., -h is no longer valid as --hostname shorthand. Also, update of `-f guard`. In 1.1-1, the `-f guard` command removed all content in the event but not the event itself, so if the event was written to file, an empty line would be the result. In 1.1-2, the event is correctly removed.
23+
2124
#### 1.1-1
2225
1.1-1 Updated kafka-clients dependency due to security vulnerability in earlier versions of the Kafka-client library used.
2326

@@ -61,6 +64,7 @@ There are input module for the following tasks:
6164
- Fetch from Elasticsearch
6265
- Static string
6366
- Static string ending with a counter starting from 1
67+
- Dynamic string with variable substitution
6468

6569
#### Read files
6670
Read a local file. (For Template files, see below)
@@ -128,14 +132,14 @@ The command line will then become:
128132
#### Receive UDP
129133
Set up a UDP server.
130134

131-
Parameters: `-i udp [-h {host}][ -p portnumber`
135+
Parameters: `-i udp [--name {host}][ -p portnumber`
132136

133137
Example: `-i udp --hostname localhost --port 5999` or `-i udp -p 5999`
134138

135139
#### Receive TCP
136140
Set up a TCP server.
137141

138-
Parameters: `-i tcp [-ih {host}] -ip portnumber`
142+
Parameters: `-i tcp [--name {host}] -ip portnumber`
139143

140144
Example: `-i tcp --hostname localhost --port 5999` or `-i tcp -p 5999`
141145

@@ -210,6 +214,18 @@ Example: `-i counter --string "Test string number:"`
210214

211215
If you want to test the generation speed, use the `null` output since that is faster than writing to the console. Add `-s true` for measurements: `java -jar target/LogGenerator-{version}.jar -i counter --string "Test:" --limit 1000000 -o null -s true`
212216

217+
#### Dynamic string with variable substitution
218+
This input item works approximately like the Template item, but you can specify a string from the command line instead of a file.
219+
220+
Parameters: `--from {the string to send, with variables} --template continuous --time-offset -10000`
221+
```
222+
--from, -fr - The string to use as a template
223+
--template, -t - [continuous, once, time:{time in ms}]
224+
--time-offset, -to - Time in ms to add or subtract from the current date, if used as a variable
225+
```
226+
227+
Example: `java -jar target/LogGenerator-1.1-1.jar -i string -t once -fr "{oneOf:A,B,C,D}" -o cmd`
228+
213229
### Output modules
214230
These are the output modules available:
215231
- Write to file
@@ -248,7 +264,7 @@ Send events with TCP.
248264

249265
Parameters: `-o tcp --hostname hostname --port port`
250266

251-
Example `-o tcp -h localhost -p 5999`
267+
Example `-o tcp --name localhost -p 5999`
252268

253269
### Write to TCP SSL
254270
Send events with encrypted TCP.
@@ -589,10 +605,9 @@ Syntax: `{pri:}`
589605
Example: `{pri:}` might be substituted by `165`
590606

591607
#### MemorySet, MemoryRecall
592-
The MemorySet will not create any output, but only save the value in the internal cache. After an expression has been
593-
evaluated, the value can be retrieved with the MemoryRecall variable.
608+
The MemorySet will not create any output, but only save the value in the internal cache. The payload of the MemorySet command will create a payload as usual. After an expression has been evaluated, the value can be retrieved with the MemoryRecall variable.
594609

595-
Syntax: `\{ms(:(?<name>[a-zA-Z0-9\-_]+))?/(?<value>.*)}`
610+
Syntax: `{ms(:(?<name>[a-zA-Z0-9\-_]+))?/(?<value>.*)}`
596611
Syntax:
597612

598613
Example:
@@ -623,7 +638,7 @@ Example: `{oneOf:{ipv4:192.168.0.0/16},{ipv4:172.16.0.0/12},{ipv4:10.0.0.0/8}}`
623638
### Read a file, add a syslog header and send the output to the console
624639
This will add a syslog header to each line in the file before printing the line.
625640

626-
`java -jar LogGenerator-{version}.jar -i file --name test.txt -f header "<{pri:}>{date:MMM dd HH:mm:ss} {oneOf:mymachine,yourmachine,localhost,{ipv4:192.168.0.0/16}} {string:a-z0-9/9}[{random:1-65535}]: " -o cmd`
641+
`java -jar LogGenerator-{version}.jar -i file --name test.txt -f header -st "<{pri:}>{date:MMM dd HH:mm:ss} {oneOf:mymachine,yourmachine,localhost,{ipv4:192.168.0.0/16}} {string:a-z0-9/9}[{random:1-65535}]: " -o cmd`
627642

628643
Example:
629644
- `<25>Dec 10 15:27:38 192.168.169.209 liiblhukp[38946]: Test row 1`

‎dependency-reduced-pom.xml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<groupId>nu.sitia.LogGenerator</groupId>
5+
<artifactId>LogGenerator</artifactId>
6+
<name>LogGenerator</name>
7+
<version>1.1-1</version>
8+
<url>http://maven.apache.org</url>
9+
<build>
10+
<plugins>
11+
<plugin>
12+
<artifactId>maven-shade-plugin</artifactId>
13+
<version>3.4.1</version>
14+
<executions>
15+
<execution>
16+
<phase>package</phase>
17+
<goals>
18+
<goal>shade</goal>
19+
</goals>
20+
<configuration>
21+
<transformers>
22+
<transformer>
23+
<mainClass>nu.sitia.loggenerator.App</mainClass>
24+
</transformer>
25+
</transformers>
26+
</configuration>
27+
</execution>
28+
</executions>
29+
<configuration />
30+
</plugin>
31+
</plugins>
32+
</build>
33+
<dependencies>
34+
<dependency>
35+
<groupId>junit</groupId>
36+
<artifactId>junit</artifactId>
37+
<version>4.13.2</version>
38+
<scope>test</scope>
39+
<exclusions>
40+
<exclusion>
41+
<artifactId>hamcrest-core</artifactId>
42+
<groupId>org.hamcrest</groupId>
43+
</exclusion>
44+
</exclusions>
45+
</dependency>
46+
</dependencies>
47+
<properties>
48+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
49+
<maven.compiler.target>11</maven.compiler.target>
50+
<maven.compiler.source>11</maven.compiler.source>
51+
</properties>
52+
</project>

‎src/main/java/nu/sitia/loggenerator/Configuration.java‎

Lines changed: 35 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ private void parseArgs(String [] args) {
170170
setEps(value);
171171
} else if (key != null && (key.equalsIgnoreCase("--statistics") || key.equalsIgnoreCase("-s"))) {
172172
// Read variables from the property file
173-
statistics = value.equalsIgnoreCase("true");
173+
statistics = value != null && value.equalsIgnoreCase("true");
174174
} else {
175175
parameters.add(key, value);
176176
}
@@ -213,28 +213,29 @@ protected KvList readConfigFromPropertyFile(String fileName) {
213213
logger.config("Reading configuration from : " + file.getAbsolutePath());
214214
try {
215215
FileInputStream input = new FileInputStream(fileName);
216-
Scanner scanner = new Scanner(input);
217-
int lineNumber = 1;
218-
while (scanner.hasNextLine()) {
219-
String line = scanner.nextLine();
220-
if (!line.startsWith("#")) {
221-
int index = line.indexOf("=");
222-
if (index < 1) {
223-
logger.fine("Disregarding line " + lineNumber + " due to missing = character");
224-
} else {
225-
String key = line.substring(0, index);
226-
String value = line.substring(index + 1);
227-
if (key.startsWith("custom.")) {
228-
customVariables.put(key.substring("custom.".length()), value);
229-
} else if (key.equalsIgnoreCase("variable-file")) {
230-
// Read variables from the property file
231-
readVariablesFromPropertyFile(value);
216+
try (Scanner scanner = new Scanner(input)) {
217+
int lineNumber = 1;
218+
while (scanner.hasNextLine()) {
219+
String line = scanner.nextLine();
220+
if (!line.startsWith("#")) {
221+
int index = line.indexOf("=");
222+
if (index < 1) {
223+
logger.fine("Disregarding line " + lineNumber + " due to missing = character");
232224
} else {
233-
list.add("--" + key, value);
225+
String key = line.substring(0, index);
226+
String value = line.substring(index + 1);
227+
if (key.startsWith("custom.")) {
228+
customVariables.put(key.substring("custom.".length()), value);
229+
} else if (key.equalsIgnoreCase("variable-file")) {
230+
// Read variables from the property file
231+
readVariablesFromPropertyFile(value);
232+
} else {
233+
list.add("--" + key, value);
234+
}
234235
}
235236
}
237+
lineNumber++;
236238
}
237-
lineNumber++;
238239
}
239240
} catch (IOException e) {
240241
throw new RuntimeException("Exception trying to load file: " + file.getAbsolutePath(), e);
@@ -262,22 +263,23 @@ protected void readVariablesFromPropertyFile(String fileName) {
262263
logger.config("Reading variables from : " + file.getAbsolutePath());
263264
try {
264265
FileInputStream input = new FileInputStream(fileName);
265-
Scanner scanner = new Scanner(input);
266-
int lineNumber = 1;
267-
while (scanner.hasNextLine()) {
268-
String line = scanner.nextLine();
269-
if (!line.startsWith("#")) {
270-
int index = line.indexOf("=");
271-
if (index < 1) {
272-
logger.fine("Disregarding line " + lineNumber + " in " + file.getAbsolutePath() + " due to missing = character");
273-
} else {
274-
String key = line.substring(0, index);
275-
String value = line.substring(index + 1);
276-
logger.finer(key + ": " + value);
277-
customVariables.put(key, value);
266+
try (Scanner scanner = new Scanner(input)) {
267+
int lineNumber = 1;
268+
while (scanner.hasNextLine()) {
269+
String line = scanner.nextLine();
270+
if (!line.startsWith("#")) {
271+
int index = line.indexOf("=");
272+
if (index < 1) {
273+
logger.fine("Disregarding line " + lineNumber + " in " + file.getAbsolutePath() + " due to missing = character");
274+
} else {
275+
String key = line.substring(0, index);
276+
String value = line.substring(index + 1);
277+
logger.finer(key + ": " + value);
278+
customVariables.put(key, value);
279+
}
278280
}
281+
lineNumber++;
279282
}
280-
lineNumber++;
281283
}
282284
} catch (FileNotFoundException e) {
283285
throw new RuntimeException(e);

‎src/main/java/nu/sitia/loggenerator/ItemProxy.java‎

Lines changed: 0 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,6 @@
2323
import nu.sitia.loggenerator.inputitems.TemplateFileInputItem;
2424
import nu.sitia.loggenerator.outputitems.OutputItem;
2525
import nu.sitia.loggenerator.templates.Template;
26-
import nu.sitia.loggenerator.templates.TemplateFactory;
2726
import nu.sitia.loggenerator.templates.TimeTemplate;
2827
import nu.sitia.loggenerator.util.LogStatistics;
2928
import sun.misc.Signal;
@@ -65,9 +64,6 @@ public class ItemProxy {
6564
*/
6665
private List<ProcessItem> gapDetectors;
6766

68-
/** The configuration object */
69-
private Configuration config;
70-
7167
private void emitMessage(String message) {
7268
List<String> result = Arrays.asList(message);
7369
for (ProcessItem item : itemList) {
@@ -87,7 +83,6 @@ private void emitMessage(String message) {
8783
* @param config The configuration
8884
*/
8985
public ItemProxy(List<ProcessItem> itemList, Configuration config) {
90-
this.config = config;
9186
this.itemList = itemList;
9287
if (config.isStatistics()) {
9388
statistics = new LogStatistics(config);
@@ -244,20 +239,4 @@ private void throttle(LogStatistics statistics) {
244239
}
245240
}
246241

247-
/**
248-
* Traverse the list of processFilters and see if any would like to be called
249-
* on exit.
250-
* @param processFilters All filters
251-
* @return A list of shutdown handlers
252-
*/
253-
private List<ShutdownHandler> getShutdownHandlers(List<ProcessFilter> processFilters) {
254-
List<ShutdownHandler> shutdownHandlerList = new LinkedList<>();
255-
processFilters.forEach(s -> {
256-
if (ShutdownHandler.class.isInstance(s)) {
257-
shutdownHandlerList.add((ShutdownHandler) s);
258-
}
259-
});
260-
return shutdownHandlerList;
261-
}
262-
263242
}

‎src/main/java/nu/sitia/loggenerator/KvList.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ public void setValue(String value) {
5151
public void add(String k, String v) {
5252
list.add(new KV(k, v));
5353
}
54-
public Iterator iterator() {
54+
public Iterator<KV> iterator() {
5555
return list.iterator();
5656
}
5757
public String get(String k) {

‎src/main/java/nu/sitia/loggenerator/ProcessItemListFactory.java‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
package nu.sitia.loggenerator;
1919

2020
import nu.sitia.loggenerator.filter.FilterItemFactory;
21-
import nu.sitia.loggenerator.filter.GapDetectionFilter;
2221
import nu.sitia.loggenerator.inputitems.InputItemFactory;
2322
import nu.sitia.loggenerator.outputitems.OutputItemFactory;
2423

@@ -37,7 +36,7 @@ public ProcessItemListFactory() {
3736
public List<ProcessItem> create(Configuration config) {
3837
List<ProcessItem> items = new LinkedList<>();
3938
ProcessItem lastItem = null;
40-
for (Iterator iterator = config.getParameters().iterator(); iterator.hasNext(); ) {
39+
for (Iterator<KvList.KV> iterator = config.getParameters().iterator(); iterator.hasNext(); ) {
4140
KvList.KV kv = (KvList.KV) iterator.next();
4241
String key = kv.getKey();
4342
String value = kv.getValue();
@@ -58,7 +57,7 @@ public List<ProcessItem> create(Configuration config) {
5857
logger.fine("Adding: " + value);
5958
} else {
6059
// Push this config to the last added item
61-
if (!lastItem.setParameter(key, value)) {
60+
if (lastItem != null && !lastItem.setParameter(key, value)) {
6261
// The item didn't accept the parameter
6362
logger.finest("Pushing: " + key + " = " + value + " to " + lastItem.getClass().getSimpleName());
6463
throw new RuntimeException("Invalid parameter: " + key + " - " + value + " " + lastItem.getClass().getSimpleName());

‎src/main/java/nu/sitia/loggenerator/filter/DropFilter.java‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919

2020

2121
import nu.sitia.loggenerator.Configuration;
22-
import nu.sitia.loggenerator.inputitems.UDPInputItem;
2322

2423
import java.util.ArrayList;
2524
import java.util.List;

‎src/main/java/nu/sitia/loggenerator/filter/GapDetectionFilter.java‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,6 @@
1818
package nu.sitia.loggenerator.filter;
1919

2020
import nu.sitia.loggenerator.Configuration;
21-
import nu.sitia.loggenerator.ShutdownHandler;
22-
import nu.sitia.loggenerator.inputitems.UDPInputItem;
2321
import nu.sitia.loggenerator.util.gapdetector.GapDetector;
2422

2523
import java.util.List;
@@ -46,7 +44,7 @@ public class GapDetectionFilter extends AbstractProcessFilter {
4644
private boolean duplicatesDetection = false;
4745

4846
/** Should we report gaps continuously? */
49-
private boolean isContinuousGapDetection;
47+
// private boolean isContinuousGapDetection;
5048

5149
/**
5250
* Create a guardFilter and set all parameters
@@ -86,7 +84,7 @@ public boolean setParameter(String key, String value) {
8684
return true;
8785
}
8886
if (key != null && (key.equalsIgnoreCase("--continuous") || key.equalsIgnoreCase("-c"))) {
89-
this.isContinuousGapDetection = value.equalsIgnoreCase("true");
87+
// this.isContinuousGapDetection = value.equalsIgnoreCase("true");
9088
logger.fine("isContinuousGapDetection " + value);
9189
return true;
9290
}

‎src/main/java/nu/sitia/loggenerator/filter/GuardFilter.java‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919

2020

2121
import nu.sitia.loggenerator.Configuration;
22-
import nu.sitia.loggenerator.inputitems.UDPInputItem;
2322

2423
import java.util.LinkedList;
2524
import java.util.List;
@@ -59,8 +58,13 @@ public boolean afterPropertiesSet() {
5958
@Override
6059
public List<String> filter(List<String> toFilter) {
6160
List<String> result = new LinkedList<>();
62-
toFilter.forEach(s -> result.add(filterLine(s)));
6361

62+
toFilter.forEach(s -> {
63+
String filteredString = filterLine(s);
64+
if (filteredString != null && filteredString.length() > 0) {
65+
result.add(filterLine(s));
66+
}
67+
});
6468
return result;
6569
}
6670

@@ -87,10 +91,10 @@ private String filterLine(String line) {
8791
* @return True iff toCheck contains a guard
8892
*/
8993
public boolean removeLine(String toCheck) {
90-
return (toCheck.startsWith(Configuration.BEGIN_TRANSACTION_TEXT)
91-
|| toCheck.startsWith(Configuration.END_TRANSACTION_TEXT)
92-
|| toCheck.startsWith(Configuration.BEGIN_FILE_TEXT)
93-
|| toCheck.startsWith(Configuration.END_FILE_TEXT));
94+
return (toCheck.contains(Configuration.BEGIN_TRANSACTION_TEXT)
95+
|| toCheck.contains(Configuration.END_TRANSACTION_TEXT)
96+
|| toCheck.contains(Configuration.BEGIN_FILE_TEXT)
97+
|| toCheck.contains(Configuration.END_FILE_TEXT));
9498
}
9599

96100
/**

0 commit comments

Comments
 (0)