77% storage cost reduction without vendor lock-in

Azure CosmosDB, Microsoft Azure Storage
Apache Avro, Kafka, Confluent Cloud

Our client, a global freight forwarder, needed to optimise its storage usage and operational costs in their data-intensive application. They handled 84.54 million requests per month, describing shipments worldwide and containing thousands of lines that were hard to separate into smaller chunks. The forwarder migrated from CosmosDB to Azure storage with VirtusLab’s help and expertise. The migration resulted in a significant reduction in costs and storage space.

The challenge: Growing data volume and storage

Due to time restrictions, large storage volume, and a limited vendor lock, the logistics company ran their operations on CosmosDB with MongoDB API. Although the CosmosDB storage solution worked well initially, the client experienced a slowly increasing number of issues with growing data volume. They encountered challenges such as unexpected costs, timeouts for full scan queries and even risk of data corruption due to the schema-less nature of the database. The forwarder commissioned VirtusLab to create a more efficient storage solution.

The solution: Azure adopting Apache Avro format

VirtusLab suggested switching from CosmoDB to Microsoft Azure Storage adopting Apache Avro format. This change helped solve storage challenges. Using Apache Avro in a way similar to how Kafka is managing data allowed the client-controlled evolution and compatibility testing of data by using schemas, which are templates defining the structure, format, and data types of stored and transmitted information. Instead of storing and sending schemas, they only sent values, reducing storage needs and expenses.

The results: Overcoming CosmosDB challenges and enhancing efficiency

Our client rid themselves of storage issues with VirtusLab’s help. They gained:

  • Modification and update of the data structure in an organised way
  • Storage reduction by 63%
  • Operational costs reduction by 77%
  • Vendor lock avoidance