Large software companies have lots of users, and the activity from those users results in high volumes of traffic. These companies also have a large surface area across the enterprise. 

There are hundreds of services and databases that are fulfilling user requests. As these requests enter the infrastructure of the enterprise, the requests travel through the different services and result in database queries, payments, and other transactions. These transactions result in the generation of log messages. The log messages tell the story of what is happening across the entire company. 

Log messages can provide valuable data for security and site reliability engineering. But analyzing a high volume of log data requires a scalable system that can account for that high volume. 

Jack Naglieri is the CEO of Panther Labs. He previously worked at Airbnb, where he helped develop a system called StreamAlert. At Airbnb, log messages are buffered into distributed queueing systems like Kafka or Kinesis, and they are written to bucket storage systems like S3. Those logs are processed by AWS Lambda functions that test the log messages for rules defined by a system operator.

Jack left Airbnb and started Panther Labs to generalize the tools he built within Airbnb and build a company around the same ideas. Jack joins the show to discuss modern logging infrastructure, his work at Airbnb, and his experience building Panther.

Contact Panther Labs at…

Sponsorship inquiries: [email protected]

The post Cloud Log Analysis with Jack Naglieri appeared first on Software Engineering Daily.