Log-A-Rhythm

Main Dashboard Page
Create your own automated warnings

Inspiration

Log aggregation allows you to gather events from various different sources into a single place so that you can search, analyse, and make sense of that data. Historically, this has been a difficult problem to solve even at a small scale in home labs; with the increase in cloud computing and also micro-service based architectures.

Log-A-Rhythm attempts to simplify this problem, we aren't the first people to try and solve this problem, however our main design goals were to abstract away the underlying tools and their complexities as well as provide a very simple to use interface allowing engineers to both analyse the problem themselves but also automate the analysis and notification procedures.

What it does

Log-A-Rhythm has client libraries that would ensure your user experience doesn't change but your logging statements are now interoperable with the rest of our tech. The user now only has to worry about the actual analysis they want to do on their logs and the rest is handled by a combination of FileBeat, LogStash and ElasticSearch, simplifying the developer experience after they've deployed their microservices.

How we built it

We've so far provided 3 client libraries for Python, C++ and Go which tightly integrate with the existing solutions present in the languages or their standard libraries. These clients ensure that the interface provided to the end programmer remains the same and is also forward and backward compatible in the case of python due to the usage of the logging modules.

These logging modules write to temporary files on device which are then forwarded to our LogStash instance by FileBeat. LogStash then performs some basic parsing and cleaning to turn the unstructured log data into structured structs extracting critical information like LogLevels, error codes, data values etc. The data is forwarded to ElasticSearch which then stores and serves the data to the frontend.

The frontend is written in Typescript using Next.js and Tailwind with a focus on simplicity and minimalism. The main design goal was to provide a feature-rich interface without clutter and visual distractions. This interface also allows developers and engineers to provide threshold values at which they are automatically notified by a transactional mail service.

Challenges we ran into

We were very unfamiliar with most of the data engineering portions of our tech stack, this was the first time any of us had used ElasticSearch or the rest of the ELK stack. A significant portion of our time was spent deciphering documentation and debugging unreadable errors in a variety of languages (Why does the ELK stack throw errors in Python, C, Ruby and Java simultaneously).

We also struggled to come up with an idea, so we lost a lot of time before working on this which could've led to a more feature complete project.

Accomplishments that we're proud of

Building something this complicated, with so many moving parts in around 12 hours!

What we learned

ElasticSearch is hard and probably wasn't the best tool to try and learn in the time frame we had. How files across different systems can be used in real time processing and data streaming Authentication for the frontend and how to use Auth providers to quickly build that up Creating real time interaction with sockets in the web app to allow for live updating.

What's next for Log-A-Rhythm

Being able to implement this for 3 languages in our time frame, makes us believe that we can further iterate and add clients for many more languages that are also prevalent in the field like Rust, Java, Scala etc.

Build out a small analytics engine while still focussing on our main goal of simplicity, which would allow us to leverage both ML and classical Statistical tools to gather even more information about our systems.