Hi,
I've been struggling to sort logs by a column but I cannot do that. How do you sort by the values in a column? In this case I want to sort by Thread ID (and if possible, then on timestamp).
Thanks!
Sorting by column
Moderator: admin
Re: Sorting by column
Hello,
In fact, you cannot sort log entries by a column at this time. It's not that we forgot to implement it, it's a choice we have made. Maybe we are wrong, but we think that seems log entries are real events, they are naturally sorted by their timestamp. Many LogMX features require that log events are sorted by timestamp: Timeline, Delta-T with a set of entries, Correlation, ... But we are always opened to new point of views.
Since you want to sort by Thread then by timestamp, maybe what you would really need is something like "group by thread" (ok, these groups may also need to be sorted...). A possible workaround for this use case is the use of the "Filter" feature. If you don't have too many different Threads, you can first display (using a filter) all entries whose Thread is X, then entries whose Thread is Y, etc... Or if you want to display only the log entries whose Thread is like "MyThreadForHTTP_.....suffix", you can also use the Filter feature with the "like" operator: "Thread like MyThreadForHTTP_.*suffix"
Feel free to let me know what you think about this issue.
Xavier
In fact, you cannot sort log entries by a column at this time. It's not that we forgot to implement it, it's a choice we have made. Maybe we are wrong, but we think that seems log entries are real events, they are naturally sorted by their timestamp. Many LogMX features require that log events are sorted by timestamp: Timeline, Delta-T with a set of entries, Correlation, ... But we are always opened to new point of views.
Since you want to sort by Thread then by timestamp, maybe what you would really need is something like "group by thread" (ok, these groups may also need to be sorted...). A possible workaround for this use case is the use of the "Filter" feature. If you don't have too many different Threads, you can first display (using a filter) all entries whose Thread is X, then entries whose Thread is Y, etc... Or if you want to display only the log entries whose Thread is like "MyThreadForHTTP_.....suffix", you can also use the Filter feature with the "like" operator: "Thread like MyThreadForHTTP_.*suffix"
Feel free to let me know what you think about this issue.
Xavier
Re: Sorting by column
Hi,
I can see the world moving on to parallel programming, thus, I would advice for a change of viewpoint.
I think LogMX would be an amazing beast against debugging parallel code ( I am working with CUDA now). In CUDA (or similar parallel paradigms), there are hundreds of threads running in parallel and your assumption of timestamp sorting is violated. Ok, you can pick a thread and look only into that but I find that quite devious and unpractical ( maybe ok for embarrassingly parallel problems ). A "sort by" and "group by" (SQL like) would be a great match for such a programming paradigm which will become more and more common with upcoming years.
Hope I showed you the need to go into that direction - if today this is not so common, it will be.
Personally, I really like LogMX but without such a feature, it renders it useless for my current working framework and I will have to start looking for other parsers unless you reconsider your choice.
Sebastian.
I can see the world moving on to parallel programming, thus, I would advice for a change of viewpoint.
I think LogMX would be an amazing beast against debugging parallel code ( I am working with CUDA now). In CUDA (or similar parallel paradigms), there are hundreds of threads running in parallel and your assumption of timestamp sorting is violated. Ok, you can pick a thread and look only into that but I find that quite devious and unpractical ( maybe ok for embarrassingly parallel problems ). A "sort by" and "group by" (SQL like) would be a great match for such a programming paradigm which will become more and more common with upcoming years.
Hope I showed you the need to go into that direction - if today this is not so common, it will be.
Personally, I really like LogMX but without such a feature, it renders it useless for my current working framework and I will have to start looking for other parsers unless you reconsider your choice.
![Rolling Eyes :roll:](./images/smilies/icon_rolleyes.gif)
Sebastian.
Re: Sorting by column
Hello Sebastian,
I agree with your analysis, tomorrow apps should get more and more parallelized, which should increase the need of such a feature. Plus, we don't like unsatisfied users. You convinced me, we will implement it shortly, trying to make it the more user-friendly we can (e.g.: when logs are not sorted by timestamp only and the user tries to use a feature that require time-sorted log entries, a message will explain that this feature is temporarily disabled because of his/her custom sorting).
I will let you know when this feature is available by posting a message in this thread.
Thank you for sharing your point of view.
Xavier
I agree with your analysis, tomorrow apps should get more and more parallelized, which should increase the need of such a feature. Plus, we don't like unsatisfied users. You convinced me, we will implement it shortly, trying to make it the more user-friendly we can (e.g.: when logs are not sorted by timestamp only and the user tries to use a feature that require time-sorted log entries, a message will explain that this feature is temporarily disabled because of his/her custom sorting).
I will let you know when this feature is available by posting a message in this thread.
Thank you for sharing your point of view.
Xavier
Re: Sorting by column
Hello Sebastian,
LogMX v5.3.1 is now released and handle log entries sorting. Entries can be sorted by any column or set of columns, up to 3 columns at the same time (e.g. by ascending Thread AND descending log level).
Xavier
LogMX v5.3.1 is now released and handle log entries sorting. Entries can be sorted by any column or set of columns, up to 3 columns at the same time (e.g. by ascending Thread AND descending log level).
Xavier
Re: Sorting by column
Hi Xavier,
Cool, I just downloaded the new version. I like the multiple column ordering as it is implemented. I did a quick test to sort by two columns and I did not get what I was expecting. Attached a printscreen with a quick test I just ran: first, I sort the log entries by Level ascending, then I sort by Thread descending. However, the Thread sorting does not affect the output at all - the entries remain sorted by the timestamp - I was expecting the two entries with thread_id ...85 and then the rest with thread_id ...84.
Am I doing something wrong or do I need to set up something beforehand? Also the third column in the ordering does not seem to affect the displayed result.
Sebastian.
Cool, I just downloaded the new version. I like the multiple column ordering as it is implemented. I did a quick test to sort by two columns and I did not get what I was expecting. Attached a printscreen with a quick test I just ran: first, I sort the log entries by Level ascending, then I sort by Thread descending. However, the Thread sorting does not affect the output at all - the entries remain sorted by the timestamp - I was expecting the two entries with thread_id ...85 and then the rest with thread_id ...84.
Am I doing something wrong or do I need to set up something beforehand? Also the third column in the ordering does not seem to affect the displayed result.
Sebastian.
- Attachments
-
- ordering.jpg (110.14 KiB) Viewed 62291 times
Re: Sorting by column
Hi Sebastian,
Thank you for your feedback, and sorry for the inconvenience. After analysis, it's in fact a bug that orders threads like "84, 85, 84", and also prevents from ordering with a 3rd column when the 1st column has similar values (here, several times "INFO"). It is now fixed, and next version v5.3.2 will include this fix, I will let you know when it is available by posting a message on this thread.
For information, this bug occurs when several entries have the same values for column "X": current implementation chooses to sort these entries by their ID, even if some other columns "Y" or more are also to be sorted => sorting by "Y" or "Z" has no effect since "X" already sorted all similar values by their ID, preventing other sorting to re-order these entries. The applied fix consists in sorting similar values by ID only if no other sorting is to be done with other column(s) after that.
Xavier
Thank you for your feedback, and sorry for the inconvenience. After analysis, it's in fact a bug that orders threads like "84, 85, 84", and also prevents from ordering with a 3rd column when the 1st column has similar values (here, several times "INFO"). It is now fixed, and next version v5.3.2 will include this fix, I will let you know when it is available by posting a message on this thread.
For information, this bug occurs when several entries have the same values for column "X": current implementation chooses to sort these entries by their ID, even if some other columns "Y" or more are also to be sorted => sorting by "Y" or "Z" has no effect since "X" already sorted all similar values by their ID, preventing other sorting to re-order these entries. The applied fix consists in sorting similar values by ID only if no other sorting is to be done with other column(s) after that.
Xavier
Re: Sorting by column
Hello Sebastian,
This bug is now fixed in LogMX v5.3.2 that was released today.
Xavier
This bug is now fixed in LogMX v5.3.2 that was released today.
Xavier
Re: Sorting by column
Hello Xavier,
about sorting User defined fields
- I have custom Java parser //tnx for example
- I have added custom filds with "entry.setUserDefinedFields(mapCustomFields);"
- all works fine, but it allways sort like text - I have Integer in cust.column
- is it possible to define column type? //Or sort type?
Petr
about sorting User defined fields
- I have custom Java parser //tnx for example
- I have added custom filds with "entry.setUserDefinedFields(mapCustomFields);"
- all works fine, but it allways sort like text - I have Integer in cust.column
- is it possible to define column type? //Or sort type?
Petr
Re: Sorting by column
Hello,
It's a very good point, thank you for the feedback.
We will add in the next release the possibility for custom Parsers to declare the type of their user-defined fields, so that, for example, if a field is declared as an Integer, its value 9 will come before 10, and not after (numeric order instead of alphanumeric order).
What do you think if we add this method in LogParser class? :
The order of the fields types would have to match the order of their names returned by getUserDefinedFields()... Maybe not really nice, but we cannot just replace this method getUserDefinedFields() for backward compatibility reason... Maybe we can just declare getUserDefinedFields() as deprecated and add a new method returning a List of UserDefinedField, which would be a new class containing the field name and its type. LogMX would handle both methods for backward compatibility for now... If you have some ideas, please let us know
Xavier
It's a very good point, thank you for the feedback.
We will add in the next release the possibility for custom Parsers to declare the type of their user-defined fields, so that, for example, if a field is declared as an Integer, its value 9 will come before 10, and not after (numeric order instead of alphanumeric order).
What do you think if we add this method in LogParser class? :
Code: Select all
public List<Class<?>> getUserDefinedFieldsClasses()
![Wink :wink:](./images/smilies/icon_e_wink.gif)
Xavier
Re: Sorting by column
Helo Petr,
LogMX v5.5.1 is now released and includes this feature
(see http://logmx.com/dev/api/com/lightysoft ... sClasses()).
Thank you for your idea!
Xavier
LogMX v5.5.1 is now released and includes this feature
![Smile :)](./images/smilies/icon_e_smile.gif)
Thank you for your idea!
Xavier