Public parser repository
Moderator: admin
-
- Posts: 2
- Joined: Wed Oct 07, 2009 4:51 pm
- Location: Austin, TX
- Contact:
Public parser repository
I need a parser for the Eclipse log file format. Is there a public parser repository where I can find a parser that supports this format?
If there is no such parser available, I'll write my own. I would be more than happy to donate my parser to a public repository so other people can use it.
Thanks,
Roy
If there is no such parser available, I'll write my own. I would be more than happy to donate my parser to a public repository so other people can use it.
Thanks,
Roy
Hello,
There's no Eclipse log file parser at this time, but you can try this Log4jPattern Parser:
(tested with Eclipse 3.4.1, workspace log file ".metadata/.log")
It handles emitter (i.e. plug-in), message, date, but not the log level, which seems to be an integer just after the emitter:
1 => info
2 => warning
4 => error
To handle these log levels, you can use this Log4jPattern Parser:
But you will have to add "1", "2", and "4" levels mapped to Info, Warning, Error to your LogMX configuration (see Tools > Options > Level)
But the best solution would be to write a Java Class Parser (http://www.logmx.com/p_parser_dev.php) to convert 1, 2, 4 to info, warning, error, and to handle "!SESSION" and "!SUBENTRY" blocks.
Let me know if you need more help to handle this format.
Xavier
There's no Eclipse log file parser at this time, but you can try this Log4jPattern Parser:
Code: Select all
!ENTRY %c %x %X %d{yyyy-MM-dd HH:mm:ss.SSS}%n!MESSAGE %m
It handles emitter (i.e. plug-in), message, date, but not the log level, which seems to be an integer just after the emitter:
1 => info
2 => warning
4 => error
To handle these log levels, you can use this Log4jPattern Parser:
Code: Select all
!ENTRY %c %p %X %d{yyyy-MM-dd HH:mm:ss.SSS}%n!MESSAGE %m
But the best solution would be to write a Java Class Parser (http://www.logmx.com/p_parser_dev.php) to convert 1, 2, 4 to info, warning, error, and to handle "!SESSION" and "!SUBENTRY" blocks.
Let me know if you need more help to handle this format.
Xavier
-
- Posts: 2
- Joined: Wed Oct 07, 2009 4:51 pm
- Location: Austin, TX
- Contact:
Yeah, I already went and wrote my own parser. Feel free to add it to your public repo:
Code: Select all
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;
/**
* @author Roy Paterson
*/
public class EclipseLogParser extends LogFileParser {
//e.g. 2009-09-15 09:34:16.000
private static final DateFormat ECLIPSE_TIMESTAMP = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
//!SESSION 2009-09-16 09:07:32.751 -----------------------------------------------
private static final Pattern SESSION = Pattern.compile("^!SESSION (.*) -----------------------------------------------$");
//!ENTRY com.smartbear.collaborator.ui.standalone 4 0 2009-09-16 09:07:41.440
private static final Pattern ENTRY = Pattern.compile("^!ENTRY (.*) (\\d) \\d (.*)$");
private ParsedEntry entry;
private StringBuilder message = new StringBuilder();
@Override
public Date getAbsoluteEntryDate(ParsedEntry entry) throws Exception {
return (Date)entry.getExtraInfo();
}
@Override
public String getParserName() {
return "Eclipse log parser";
}
@Override
public Date getRelativeEntryDate(ParsedEntry entry) throws Exception {
return getAbsoluteEntryDate(entry);
}
@Override
public String getSupportedFileType() {
return "Eclipse logs";
}
@Override
protected void parseLine(String line) throws Exception {
//end of file
if (line == null) {
recordPreviousEntryIfExists();
return;
}
Matcher matcher;
//check for SESSION
matcher = SESSION.matcher(line);
if (matcher.matches()) {
recordPreviousEntryIfExists();
entry = createNewEntry();
entry.setEmitter("SESSION");
entry.setDate(matcher.group(1));
entry.setExtraInfo(ECLIPSE_TIMESTAMP.parse(matcher.group(1)));
entry.setLevel("INFO");
message.setLength(0); //reset message buffer
return;
}
//check for ENTRY
matcher = ENTRY.matcher(line);
if (matcher.matches()) {
recordPreviousEntryIfExists();
entry = createNewEntry();
entry.setEmitter(matcher.group(1));
entry.setLevel(toLogMxLevel(Integer.parseInt(matcher.group(2))));
entry.setDate(matcher.group(3));
entry.setExtraInfo(ECLIPSE_TIMESTAMP.parse(matcher.group(3)));
message.setLength(0); //reset message buffer
return;
}
//accumulate message
message.append(line).append("\n");
}
/**
* Send to LogMX the current parsed log entry
* @throws Exception
*/
private void recordPreviousEntryIfExists() throws Exception {
if (entry != null) {
entry.setMessage(message.toString());
addEntry(entry);
}
}
private static String toLogMxLevel(int statusCode) {
switch(statusCode) {
case 0: //IStatus.OK
return "INFO";
case 1: //IStatus.INFO
return "INFO";
case 2: //IStatus.WARNING
return "WARNING";
case 4: //IStatus.ERROR
return "ERROR";
case 8: //IStatus.CANCEL
return "WARNING";
default:
return "ERROR";
}
}
}
Thank you very much, well done
May I suggest an optimization: using reluctant quantifier ".*?" instead of greedy quantifier ".*" will greatly improve regular expression processing. I think it can be used here without problem (reluctant performs a "minimal matching" compared to greedy that will try to match all characters in string)
May I suggest an optimization: using reluctant quantifier ".*?" instead of greedy quantifier ".*" will greatly improve regular expression processing. I think it can be used here without problem (reluctant performs a "minimal matching" compared to greedy that will try to match all characters in string)
Re: Public parser repository
Here is my simple parser to parse fixed length log files genereted by CAN Analyzer CanKing:
Code: Select all
package sample.parser;
import java.text.DecimalFormatSymbols;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.text.DecimalFormat;
import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;
/**
*
* @author kramera
*
* Parser for CanKing log files of following format (fixed positions)
0 0C49020D X 6 30 02 00 00 00 02 0..... 34769.328890 R
0 0C490D02 X 8 20 63 75 24 38 20 36 35 cu$8 65 34769.329720 R
0 0C490D02 X 8 21 35 33 35 20 2D 31 04 !535 -1. 34769.330360 R
0 0C490D02 X 2 22 FA ". 34769.330810 R
0 0C49020D X 6 30 00 00 02 00 02 0..... 34769.333690 R
0 00000342 8 4A 56 00 00 00 00 00 00 JV...... 34769.646650 R
0 000002C2 8 4A 56 00 00 00 00 00 00 JV...... 34769.647550 R
0 00000342 8 42 47 00 40 00 00 00 00 BG.@.... 34769.748600 R
0 000002C2 8 42 47 00 00 00 00 00 00 BG...... 34769.748990 R
0 0C44020D X 8 10 11 52 65 61 64 41 63 ..ReadAc 34770.010490 R
0 0C440D02 X 6 30 01 00 00 00 01 0..... 34770.011130 R
0 0C44020D X 8 20 63 75 42 79 74 65 24 cuByte$ 34770.011770 R
0 0C44020D X 5 21 31 33 05 14 !13.. 34770.012600 R
0 0C45020D X 8 10 0D 52 65 61 64 41 63 ..ReadAc 34770.013880 R
0 0C46020D X 8 10 0D 49 73 41 63 63 75 ..IsAccu 34770.014590 R
0 0C440D02 X 6 30 00 00 02 00 01 0..... 34770.017850 R
*/
public class CanKingParser extends LogFileParser {
private DecimalFormatSymbols dc;
private DecimalFormat nf;// = new DecimalFormat("#.#",);
/** Key of user-defined field "timestamp" */
private static final String EXTRA_FIELD_KEY_CAN_ID = "Object Id";
private static final String EXTRA_FIELD_KEY_CAN_DATA_HEX = "Data Hex";
private static final String EXTRA_FIELD_KEY_CAN_DATA_LENGTH = "Data Length";
private static final String EXTRA_FIELD_KEY_CAN_DESTINATION = "Destination";
private static final String EXTRA_FIELD_KEY_CAN_EXTENDED = "Extended";
public CanKingParser()
{
dc = new DecimalFormatSymbols();
dc.setDecimalSeparator('.');
nf = new DecimalFormat("#.#",dc);
}
/** User-defined fields names (here, only one) */
private static final List<String> EXTRA_FIELDS_KEYS = Arrays
.asList(EXTRA_FIELD_KEY_CAN_ID,
EXTRA_FIELD_KEY_CAN_DATA_HEX,
EXTRA_FIELD_KEY_CAN_DATA_LENGTH,
EXTRA_FIELD_KEY_CAN_DESTINATION,
EXTRA_FIELD_KEY_CAN_EXTENDED);
private ParsedEntry entry = null;
@Override
public Date getAbsoluteEntryDate(ParsedEntry arg0) throws Exception {
//return DATE_FORMAT.parse(arg0.getDate()); // (the right-part "T0+..." will be ignored by the formatter)
double n = nf.parse(arg0.getDate()).doubleValue();
return new Date((long)(n*1000.0));
}
@Override
public String getParserName() {
return "CanKing Log Parser V2";
}
@Override
public Date getRelativeEntryDate(ParsedEntry arg0) throws Exception {
return getAbsoluteEntryDate(arg0);
}
@Override
public String getSupportedFileType() {
return "CanKing log Files";
}
/**
* Returns the ordered list of user-defined fields to display (given by their key), for each entry.
* @see com.lightysoft.logmx.mgr.LogFileParser#getUserDefinedFields()
*/
@Override
public List<String> getUserDefinedFields() {
return EXTRA_FIELDS_KEYS;
}
@Override
protected void parseLine(String arg0) throws Exception {
if (arg0 == null)
return;
if (arg0.length() != 84)
return;
entry = createNewEntry();
assert(entry!=null);
if (entry != null)
{
String extended = arg0.substring(15, 16).trim();
boolean extSyntax = extended.compareTo("X")==0;
String canId = arg0.substring(6, 14);
String timeStamp=arg0.substring(70, 82);
entry.setDate(timeStamp);
/*
double n = nf.parse(timeStamp).doubleValue();
Double db = new Double(n);
String tmp = db.toString();
entry.setThread(tmp);
*/
if (extSyntax)
entry.setLevel("CAN_EXT");
else
entry.setLevel("CAN_NORM");
if (extSyntax)
entry.setEmitter(canId.substring(6,8));
else
entry.setEmitter(canId);
entry.setMessage(arg0.substring(58, 66));
entry.setUserDefinedFields(new HashMap<String, Object>(3));
entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_ID,canId );
entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_DATA_HEX, arg0.substring(26, 56));
entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_DATA_LENGTH, arg0.substring(23, 24));
if (extSyntax)
entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_DESTINATION, canId.substring(4,6));
entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_CAN_EXTENDED, extended);
addEntry(entry);
}
}
}
Re: Public parser repository
Hello alexk195,
Thank you for your contribution we have added your parser to our internal repository.
Our public parser repository will be on-line as soon as we have collected a few more parsers.
Xavier.
Thank you for your contribution we have added your parser to our internal repository.
Our public parser repository will be on-line as soon as we have collected a few more parsers.
Xavier.
Re: Public parser repository
Here my another custom Parser for parsing linux kernel logs, dmesg output etc. It searches for keywords like "error","panic", "warning" for appropriate level. Still needs proper time stamp parsing but works in general.
@admin/developer: Some feature to disable columns in custom parser would be nice! E.g. Don't show Emitter column, if every entry has null Emitter.
@admin/developer: Some feature to disable columns in custom parser would be nice! E.g. Don't show Emitter column, if every entry has null Emitter.
Code: Select all
package sample.parser;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;
/**
*
* @author Alexander Kramer
*
* Parser for Kernel logs, detects words "error","panic","warning" in messages to set the level.
* TODO: Parsing timestamp doesn't work.
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2346369
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2373697
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2401089
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2428577
*/
public class KernelLogParser extends LogFileParser {
protected SimpleDateFormat dateFormat = new SimpleDateFormat("MMM dd HH:mm:ss");
public KernelLogParser()
{
}
/**
* Custom columns as String constants
*/
private static final String EXTRA_FIELD_KEY_HOST_NAME = "Host";
/**
* Put all custom fields in an const array
*/
private static final List<String> EXTRA_FIELDS_KEYS = Arrays
.asList(EXTRA_FIELD_KEY_HOST_NAME
);
private ParsedEntry entry = null;
@Override
public Date getAbsoluteEntryDate(ParsedEntry arg0) throws Exception {
Date date = new Date();
parseDateTime(arg0.getDate(), date);
return date;
}
@Override
public String getParserName() {
return "Kernel Log Parser V1";
}
@Override
public Date getRelativeEntryDate(ParsedEntry arg0) throws Exception {
return getAbsoluteEntryDate(arg0);
}
@Override
public String getSupportedFileType() {
return "Kernel logs";
}
/**
* Returns the ordered list of user-defined fields to display (given by their key), for each entry.
* @see com.lightysoft.logmx.mgr.LogFileParser#getUserDefinedFields()
*/
@Override
public List<String> getUserDefinedFields() {
return EXTRA_FIELDS_KEYS;
}
/// returns true if timestamp is in proper format
protected boolean parseDateTime(String dateTime, Date date)
{
String[] fields = dateTime.split(" ");
if (fields.length != 3)
return false;
String[] timeFields = fields[2].split(":");
if (timeFields.length != 3)
return false;
try
{
date = dateFormat.parse(dateTime);
}catch(Exception e)
{
// SimpleDateFormat cannot parse it, TODO: define format or parse by itself
// return false;
}
return true;
}
@Override
protected void parseLine(String arg0) throws Exception {
if (arg0 == null)
return;
entry = createNewEntry();
if (entry != null)
{
String dateTimeString = arg0.substring(0, 16).trim();
Date date = new Date();
// quick check of format
if (!parseDateTime(dateTimeString, date))
return;
entry.setDate(dateTimeString);
String exclTimeStr = arg0.substring(16).trim();
int spaceIndex = exclTimeStr.indexOf(' ');
//additional check for format
if (spaceIndex<0) return;
String hostName = exclTimeStr.substring(0,spaceIndex).trim();
entry.setUserDefinedFields(new HashMap<String, Object>());
entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_HOST_NAME, hostName);
String message = exclTimeStr.substring(spaceIndex).trim();
entry.setMessage(message);
// defining levels, search for keywords
String messageLowerCase = message.toLowerCase();
if ((messageLowerCase.indexOf("error")>=0)|| messageLowerCase.indexOf("panic")>=0)
entry.setLevel("ERROR");
else
if (messageLowerCase.indexOf("warning")>=0)
entry.setLevel("WARNING");
else
entry.setLevel("TRACE");
// We don't use them:
entry.setEmitter(null);
entry.setThread(null);
addEntry(entry);
}
}
}
Re: Public parser repository
Hello,
Thank you!
Here is a version with Timestamp parsing.
In your version, Timestamp parsing didn't work because in the "parseDateTime()" method, the new value for the "date" parameter was lost once this method exited (in Java, all parameters are passed by value and not by reference).
Note that the timestamp year is missing in this log file format, so LogMX considers it to be 1970.
Also note that the timestamp parsing depends on the Locale specified for this Parser (set in "Tools" > "Options" > "Parsers" > "Kernel logs"). For example, to parse "Aug" as the 8th month of the year, your must specify an English Locale for this Parser.
Thank you!
Here is a version with Timestamp parsing.
In your version, Timestamp parsing didn't work because in the "parseDateTime()" method, the new value for the "date" parameter was lost once this method exited (in Java, all parameters are passed by value and not by reference).
Note that the timestamp year is missing in this log file format, so LogMX considers it to be 1970.
Also note that the timestamp parsing depends on the Locale specified for this Parser (set in "Tools" > "Options" > "Parsers" > "Kernel logs"). For example, to parse "Aug" as the 8th month of the year, your must specify an English Locale for this Parser.
Code: Select all
package sample.parser;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Arrays;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import com.lightysoft.logmx.business.ParsedEntry;
import com.lightysoft.logmx.mgr.LogFileParser;
/**
*
* @author Alexander Kramer
*
* Parser for Kernel logs, detects words "error","panic","warning" in messages to set the level.
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2346369
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2373697
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2401089
Aug 10 07:27:52 LLS061101302 kernel: lost page write due to I/O error on mmcblk1p1
Aug 10 07:27:52 LLS061101302 kernel: end_request: I/O error, dev mmcblk1, sector 2428577
*/
public class KernelLogParser extends LogFileParser {
protected SimpleDateFormat dateFormat = null;
/** Pattern for entry begin */
private final static Pattern ENTRY_BEGIN_PATTERN = Pattern
.compile("^(\\S+?\\s+?\\d{1,2}\\s+?\\d{2}:\\d{2}:\\d{2})\\s+?(\\S+?)\\s+(.*)$");
public KernelLogParser() {
}
/**
* Custom columns as String constants
*/
private static final String EXTRA_FIELD_KEY_HOST_NAME = "Host";
/**
* Put all custom fields in an const array
*/
private static final List<String> EXTRA_FIELDS_KEYS = Arrays
.asList(EXTRA_FIELD_KEY_HOST_NAME);
private ParsedEntry entry = null;
@Override
public Date getAbsoluteEntryDate(ParsedEntry arg0) throws Exception {
return parseDateTime(arg0.getDate());
}
@Override
public Date getRelativeEntryDate(ParsedEntry arg0) throws Exception {
return getAbsoluteEntryDate(arg0);
}
@Override
public String getParserName() {
return "Kernel Log Parser V2";
}
@Override
public String getSupportedFileType() {
return "Kernel logs";
}
/**
* Returns the ordered list of user-defined fields to display (given by their key), for each entry.
* @see com.lightysoft.logmx.mgr.LogFileParser#getUserDefinedFields()
*/
@Override
public List<String> getUserDefinedFields() {
return EXTRA_FIELDS_KEYS;
}
/// returns a non-null Date if timestamp is in proper format
protected Date parseDateTime(String dateTime) {
if (dateFormat == null) {
// Now create the date formatter using the right Locale
// (method "getLocale()" can't be called from the constructor)
dateFormat = new SimpleDateFormat("MMM d HH:mm:ss", getLocale());
}
try {
return dateFormat.parse(dateTime);
} catch (ParseException e) {
// Invalid format
return null;
}
}
@Override
protected void parseLine(String line) throws Exception {
if (line == null)
return;
// Check line format
Matcher matcher = ENTRY_BEGIN_PATTERN.matcher(line);
if (!matcher.matches()) {
return;
}
entry = createNewEntry();
entry.setDate(matcher.group(1));
String hostName = matcher.group(2);
entry.setUserDefinedFields(new HashMap<String, Object>());
entry.getUserDefinedFields().put(EXTRA_FIELD_KEY_HOST_NAME, hostName);
String message = matcher.group(3).trim();
entry.setMessage(message);
// defining levels, search for keywords
String messageLowerCase = message.toLowerCase();
if ((messageLowerCase.indexOf("error") >= 0) || messageLowerCase.indexOf("panic") >= 0)
entry.setLevel("ERROR");
else if (messageLowerCase.indexOf("warning") >= 0)
entry.setLevel("WARNING");
else
entry.setLevel("TRACE");
// We don't use them:
entry.setEmitter(null);
entry.setThread(null);
addEntry(entry);
}
}
Re: Public parser repository
Hello
Is there a public parser repository now?
Is there an example somewhere, how to read Websphere SystemOut.log and SystemErr.log, the websphere appserver logfiles?
Regards
Matthias
Is there a public parser repository now?
Is there an example somewhere, how to read Websphere SystemOut.log and SystemErr.log, the websphere appserver logfiles?
Regards
Matthias
Re: Public parser repository
Hello,
The public parser repository is not available yet, but here is a parser for Websphere logs:
http://www.logmx.com/downloads/forum/We ... Parser.zip
This ZIP file contains the LogMX parser, its source file, a log file example, and an info file about this log format (this package comes from our "Private parser repository", that should become "Public" in the future ).
To test it, simply copy the class file in LogMX directory "parsers\classes\sample\parser" (see http://www.logmx.com/p_parser_dev.php for more details on Parsers development)
Xavier.
The public parser repository is not available yet, but here is a parser for Websphere logs:
http://www.logmx.com/downloads/forum/We ... Parser.zip
This ZIP file contains the LogMX parser, its source file, a log file example, and an info file about this log format (this package comes from our "Private parser repository", that should become "Public" in the future ).
To test it, simply copy the class file in LogMX directory "parsers\classes\sample\parser" (see http://www.logmx.com/p_parser_dev.php for more details on Parsers development)
Xavier.
Re: Public parser repository
Hi Xavier
Thank you very much for the fast answer. I will try your parser.
Regards
Matthias
Thank you very much for the fast answer. I will try your parser.
Regards
Matthias