How to fix separator found on tag ?

Notes, tips, and other usefull things on how to use LogMX

Moderator: admin

Post Reply
Jack
Posts: 7
Joined: Fri Mar 14, 2014 2:46 pm

How to fix separator found on tag ?

Post by Jack » Mon Nov 03, 2014 4:32 pm

I try to find a solution for the following problem:
I have a complex log4j pattern with 14 tags, with the '-' character used as separator.
Unfortunately, the separator is not well chosen, as a tag can contain this separator :roll:
I can't change the log4j pattern, and I try to find a solution to fix this problem.
Here is my pattern :

Code: Select all

%d [%p] %t'-%mx{codeCentre}-%mx{loginUtilisateur}-%mx{idFournisseur}-%mx{energie}-%mx{numeroPdl}-%mx{service}-%mx{version}-(%c{1}.%M():%L) %m%n
Here are different solutions I think/try :

1)
The tag containing the separator (loginUtilisateur) contain an email, so always with the '@' character. As the '-' is always on the first part of the email (before the @), I try to separate this tag in 2 with adding a @ as a separator like that :

Code: Select all

%d [%p] %t'-%mx{codeCentre}-%mx{loginUtilisateur}@%mx{loginUtilisateur2}-%mx{idFournisseur}-%mx{energie}-%mx{numeroPdl}-%mx{service}-%mx{version}-(%c{1}.%M():%L) %m%n
Unfortunately, this tag can be also empty, with no @, so it was not working.

2)
Writing a new parser.
As my pattern is long and complex, I really don't want to rewrite all the parser!
I saw it's possible to call the LogMX parser to don't have to re-write the all parser (huge job!), but as classes are obfuscated to avoid a copy of LogMX code, I'm not sure it's a really confident solution. I don't know the class name to call, and it can change for the next LogMX release...

3)
Writing a new manager.
I thought writing a new simple manager to analyze each entry and replace the separator '-' in the tag "loginUtilisateur" submit the log to the parser.
My files are all local, and it will be the LocalFileManager with a kind of pre-code.
But I'm not sure I can do that in a manager...


For me, the most reliable solution seems 3) to write a new manager, but I'm not sure it's possible.
What do you think? Any idea?

admin
Site Admin
Posts: 418
Joined: Sun Dec 17, 2006 10:30 pm

Re: How to fix separator found on tag ?

Post by admin » Mon Nov 03, 2014 5:40 pm

Hello Jack,

To make your current Log4j pattern parser work, you simply have to click on the button "Tags settings" in the Parser creation wizard, then uncheck "Match minimum characters" for "loginUtilisateur" tag. Here is a screenshot:

Image

Just for fun, another solution would be to use a Regular Expression Parser instead (a bit more powerful):
(.+) \[(.*?)\] (.*?)'-(.*?)-(.*)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?)-\((.*?)\.(.*?)\(\):(.*?)\) (.*)

Image

Anyway, here are answers to your issues (for your information):
  • Solution 1) Impossible to make it work in your case
  • Solution 2) Possible, but above solutions are simpler in this case. Note that there is no obfuscation issue here: you simply write your Java Class Parser using LogMX API. Writing a Java Parser is not a hack in LogMX, it's a feature ;)
  • Solution 3) Maybe possible, but way too complex compared to above solutions

Jack
Posts: 7
Joined: Fri Mar 14, 2014 2:46 pm

Re: How to fix separator found on tag ?

Post by Jack » Tue Nov 04, 2014 10:22 am

Thanks for the quick answer!

I know the "match minimum characters" feature, and I already try it. But I can't manage it working in my case, I think because my line are not all complete in the same way, even if they are made by a log4j file :(

Here are an example of few lines I can't make it working:

Code: Select all

2014-06-29 22:30:38,649 [DEBUG] [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'--------(SessionListener.sessionCreated():56) [session id : dsfgdsfgdsfgdsfg!13123213!65465654564]
2014-06-29 22:30:38,652 [INFO ] [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'-145-bill@mail.com-04-E-14577134518385-getinfos-11-229-(SOAPMessageHandler.getSoapHeader():148) Identifiant de la session ALSB : sdfgdfg-sdfgdf.dsfgdfg.4e96 [session id : dsfgdsfgdsfgdsfg!13123213!65465654564]
2014-06-29 22:30:38,653 [INFO ] [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'-145-bill-bob@mail.com-04-E-14577134518385-getinfos-11-229-(SOAPMessageHandler.executeMethodRequest():39) SOAP MESSAGE request : [session id : dsfgdsfgdsfgdsfg!13123213!65465654564]
2014-06-29 22:31:15,623 [INFO ] [ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'-063-bob@mail.com-04-E-06365412399197-getinfos-11-229-(SOAPMessageHandler.executeMethodRequest():39) SOAP MESSAGE request : [session id : sdfgdsfgdfgdsfg!13123213!65465654564]
And it's not better with the regex expression. Or maybe I just don't succeed to make it working, as regex it's not really my cup of tea...

Another point to consider is that my log files are huge (sometimes more than 400Mo). Actually it works with the configuration of MAX_MEMORY=4000M. I just have to be patient to load the file, it's not a problem. But I wonder if I finally have to implement a java class parser if it can be a problem.

admin
Site Admin
Posts: 418
Joined: Sun Dec 17, 2006 10:30 pm

Re: How to fix separator found on tag ?

Post by admin » Tue Nov 04, 2014 4:27 pm

Hello Jack,

The Log4j Pattern Parser works for me for these 4 new lines of log (with "match minimum characters" set only for "loginUtilisateur"):
Image

The Regex Parser does too, with a small tuning I made:

Code: Select all

(.+?) \[\s*(\S+?)\s*\] (.*?)'-(.*?)-(.*)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?)-\((.*?)\.(.*?)\(\):(.*?)\) (.*)
Image

Maybe you missed something when creating the Log4j Parser. For your convenience, I've created an export of these two Parsers I've created: http://www.logmx.com/downloads/forum/lo ... ers.export. You can import these parsers using "Tools" > "Options" > "General" > button "Import options..."

Please let me know if you still have issues.

Xavier

Jack
Posts: 7
Joined: Fri Mar 14, 2014 2:46 pm

Re: How to fix separator found on tag ?

Post by Jack » Wed Nov 05, 2014 9:10 am

Thanks for you effort Xavier!
But in fact, it still doesn't work :(

With the regex parser you give me, all the fields are shift after 'loginUtilisateur'. idFournisseur show Energie and so on.
Same for log4j pattern parser
But I understand why: It's because when the 'version' field is present, it also contain a '-' character :( In the few lines I give you, the version is in fact "11-229" for the last 3 line.

I try to uncheck also "match minimum characters" for version, but it doesn't work. It's understandable why.

I think there is no simple solution with 2 fields containing potentially the separator. If I'm wrong, tell me!
But it's completely understandable for me that it's not possible to match field if there are also separators in at least 2 fields.

So actually I see 2 solutions:
- Forget line like first line (who doesn't show me many information anyway) with adding a new field to match the second part of version
- Develop a specific parser for my case.

admin
Site Admin
Posts: 418
Joined: Sun Dec 17, 2006 10:30 pm

Re: How to fix separator found on tag ?

Post by admin » Wed Nov 05, 2014 5:05 pm

You're right, if at least two fields may contain the separator, no Regex or Pattern will be able to make a difference between "john-doe@mail.com" / "63", and "john" / "doe@mail.com-63" for example.

As for a suitable solution in your (difficult) case, I'm not sure that ignoring lines like "[...]------[...]" will help, because the empty string "" is a string like any other strings (will be treated as any other field value). So you may want to write your own Parser and add some logical tests in it, depending on your business objects rules (e.g. recognizing e-mail addresses, versions, ...). Note that if you have purchased a Pro version, you can reach the Technical Support at support@logmx.com, so that we can write your Parser according to your Specifications (or help you writing it).

Xavier

Jack
Posts: 7
Joined: Fri Mar 14, 2014 2:46 pm

Re: How to fix separator found on tag ?

Post by Jack » Thu Nov 06, 2014 9:33 am

By forgetting the first line I mean make a log4j pattern parser not matching these kind of line by adding a new field.
With the following pattern, replacing "-%mx{version}" by "-%mx{vx}-%mx{vy}", I get correctly other lines (3 last lines of my pattern test):

Code: Select all

%d [%p] %t'-%mx{centre}-%mx{loginUtilisateur}-%mx{fournisseur}-%mx{energie}-%mx{numeroPdl}-%mx{service}-%mx{vx}-%mx{vy}-(%c{1}.%M():%L) %m%n
I have a Pro version :) but I will try first with this pattern if it's enough for me. I will see after if I need to write my own parser with my specific business rules.

Thanks for the very pertinent help!

Post Reply