May 14, 2014

q - Text as Database

Nice software, that allows treat text as database - q Text as data.

Shortly, it allows to perform SQL-like queries in tabular text. For example, we have some file.txt with following data:
1 646524 some_text
5 878060 some_other_text
9 676876 some_text

And if q is installed we can perform SQL query like this (where cN is column number):
q "SELECT sum(c1), c3 FROM file.txt WHERE c2 > 600000 GROUP BY c3 ORDER BY 2"

The output will be:
5 some_other_text
10 some_text

Very simple (for those who knows SQL) and convenient way to work with big files with tabular data.

In our project we use it to obtain data about slow java methods in our application: once a day automated script extracts data about execution time of methods from logs and then aggregates executed text with q to group methods and show the slowest and most frequent of them (and then we use AWK to show final result in HTML table).

So, It's good to know about this tool for those, who encounter with tabular texts.

No comments:

Post a Comment