Introduction

In modern cloud environments, traditional approaches for storing logs in isolated systems have become inadequate. As distributed software systems become more common, where different components run across multiple services and regions, it is essential to continuously collect and forward both system and application logs to a centralized location for in-depth analysis. These logs play an important role in debugging, performance monitoring, and ensuring the overall health and reliability of the infrastructure. 

In the AWS cloud environment, many such components of the distributed software system are still hosted on Amazon EC2 instances and use an agent-based approach to transmit system and application logs to a centralized service, where this data is ingested and stored for further use by observability platforms. While observability improves operational insight and system reliability, it also increases the cost of data ingestion and long-term storage. Therefore, organizations must maintain a careful balance between observability depth and the financial sustainability of the platform. Selecting a resilient, scalable, and cost-effective ingestion and storage solution has become an important element of any observability strategy, especially when the platform is being used at enterprise scale.