The Perks of Online vs Local Logging

In today’s age of big data cruising at massive volumes and breakneck speeds, if (God forbid) that production environment goes down on a Saturday night, you will want a proper logging solution to be able to mend the problem as fast and efficiently as possible. That means you will need tools that will show you what happened and at what time, instead of burning the midnight oil doing your best guesswork.

In this article we’ll try to look at (and answer):

• What is a logging solution
• Why use logging
• X vs Y

What is a logging solution?

A logging solution, or a Log Management System (LMS) is a software solution that gathers, sorts and stores log data and event logs from a variety of sources in one centralized location. Log management software systems allow IT teams, DevOps and SecOps professionals to establish a single point from which to access all relevant network and application data.

Yet, if you’re here, you probably already knew that.

It is quite difficult for someone in this field (such as myself) to write a piece on this topic without seeming preferential to one solution as opposed to another instead of an unbiased overview, as there are many options to choose from, such as ELK, Splunk, and DataDog to name a few.

In terms of what you want from your logging solution, modern log management and analysis solutions include the following key capabilities:

• Aggregation – the ability to collect and ship logs from multiple data sources.
• Processing – the ability to transform log messages into meaningful data for easier analysis.
• Storage – the ability to store data for extended time periods to allow for monitoring, trend analysis, and security use cases.
• Analysis – the ability to dissect the data by querying it and creating visualizations and dashboards on top of it.

The widely popular ELK stack, composed of Elasticsearch, Logstash and Kibana – with Elasticsearch being the search and analysis engine, Logstash being the aggregator and Kibana being the visualization tool – are by no means the only ways to be.

However, a nice perk of the aforementioned (beside the fact that it is open source) is that the key components can be swapped in and out if for example one prefers a specific aggregator over another. With enough jerry-rigging, the crafted solution will cater to your needs.

Having said that, let’s move on to our next question.

Why use logging?

Don’t be silly. Having a logging solution is paramount for effective troubleshooting when either something inevitably goes wrong, or when you want to check your work in progress for the appropriate behavior.

Of course, this can vary depending on the situation, as a logging solution would best fit a use case better than other. Having said that, any of the examples previously given would still be a superior choice to relying purely on text logs for a myriad of reasons, chief among them being accessibility, presentability and a means of filtering data.

If time is of the essence when an issue appears, having the ability to quickly view logs from a designated platform becomes critical. Add to this the ability to narrow down a specific time for the occurred event, isolate segments of data from others while having said data displayed comprehensively. All of this make the experience immensely more desirable as opposed to sifting through a text file with your old friend CTRL + F in hopes of finding out what happened.

The advantages of logging software for fintechs

Modern fintech companies are cloud-native, data-driven organizations that depend on log data for a number of use cases, such as troubleshooting cloud services and infrastructure. However, there are several other benefits for fintechs in using logging software:

1. Auditing and compliance: Logging software can help financial technology companies comply with regulatory requirements by providing a comprehensive record of all transactions, user activity, and system changes.
2. Fraud detection: Logging software can be used to detect and prevent fraudulent activity by analyzing patterns in user behavior and transactions.
3. Improved security: Logging software can help fintech companies identify and respond to security threats by providing visibility into system and network activity.
4. Performance monitoring: Logging software can help fintech companies monitor the performance of their systems and identify any issues that may be impacting user experience.
5. Debugging: Logging software can help developers quickly identify and fix errors in their code by providing detailed information about system activity.

Some fintech companies today also build their own complex data pipelines to aggregate and ingest log data from cloud-based applications and services, normalize and transform those logs, and move them to traditional log analytics solutions like Elasticsearch, Splunk, or Sumo Logic.

X vs Y

Now, although all the above sound swell, the question remains: ‘what should I use?’. Because, let’s face it, in terms of options we are truly spoiled for choice.

There are plenty of sales pitches and versus articles to be found for each logging solution, each differing in many ways from each other, going from being open source and customizable, to old but reliable. Some might require a premium option to be paid, yet you will get everything neatly packaged.

Spoiled for choice indeed. My advice? Do a bit of light reading on what might fit the bill for you instead of relying on that old text log file. In the end, you might end up finding a drop-fit solution for your use case or you might need to integrate different workflows yourself. But what really matters, rather than tooling and frameworks, is valuing the quality-of-life improvements an integrated logging system brings, on top of visibly increased stability for your products. All in a day’s work.