JIYIK CN >

Log files contain information about events that occurred during the operation of a software system or application. These events include errors, requests made by users, bugs, etc. Developers can further scan these usage details to find potential problems with the system, implement newer and better solutions, and improve the overall design. Log files can reveal a lot of information about the security of the system, which helps developers improve the system or application.

Typically, entries in log files have a format or pattern. For example, a software system may have a format that prints three things: a timestamp, the log message, and the message type. These formats can contain any amount of information in a nicely formatted text for readability and management purposes.

To perform analysis on these log files, one can consider using any programming language. But this article will specifically discuss how to parse such log files using Python. Nevertheless, the theory behind the process is the same for all programming languages. One can easily convert Python code into any other programming language to perform the required task.

Parsing log files in Python

As mentioned above, the entries in the log file have a specific format. This means that we can utilize this format to parse the information written in the log file line by line. Let's try to understand this with an example.

Consider the following log format for a web application. It has four important details, namely, date and time or timestamp ( yyyy-mm-dd hh:mm:ssformat), access URL, type of log message (success, error, etc.), and the log message.

DateTime | URL | Log - Type | Log

Now, consider a file containing logs in the above format log.txt. log.txtThe file looks like this.

2021-10-26 10:26:44 | https://website.com/home | SUCCESS | Message
2021-10-26 10:26:54 | https://website.com/about | SUCCESS | Message
2021-10-26 10:27:01 | https://website.com/page | ERROR | Message
2021-10-26 10:27:03 | https://website.com/user/me | SUCCESS | Message
2021-10-26 10:27:04 | https://website.com/settings/ | ERROR | Message
...

The following Python code will read this log file and store the information in a dictionary. The variable orderstores all the dictionary keys in the same order as a single log. Since the log form has a |, we can use it to split the log string into elements and further store the elements we like.

import json

file_name = "log.txt"
file = open(file_name, "r")
data = []
order = ["date", "url", "type", "message"]

for line in file.readlines():
    details = line.split("|")
    details = [x.strip() for x in details]
    structure = {key: value for key, value in zip(order, details)}
    data.append(structure)

for entry in data:
    print(json.dumps(entry, indent=4))

输出

{
    "date": "2021-10-20 10:26:44",
    "url": "https://website.com/home",
    "type": "SUCCESS",
    "message": "Message",
}
{
    "date": "2021-10-20 10:26:54",
    "url": "https://website.com/about",
    "type": "SUCCESS",
    "message": "Message",
}
{
    "date": "2021-10-20 10:27:01",
    "url": "https://website.com/page",
    "type": "ERROR",
    "message": "Message",
}
{
    "date": "2021-10-20 10:27:03",
    "url": "https://website.com/user/me",
    "type": "SUCCESS",
    "message": "Message",
}
{
    "date": "2021-10-20 10:27:04",
    "url": "https://website.com/settings/",
    "type": "ERROR",
    "message": "Message",
}

After reading the information, we can perform any further actions on it. We can store it in a database for future analysis, import NumPyand Matplotliband draw some graphs to understand the information graphically. Filter ERRORlogs with the tag and scan for errors faced by users, or watch out for some suspicious activities or security breaches like spam or unauthorized access. The opportunities are endless, depending on what the developer or data scientist is trying to learn from the data obtained.