Event-Driven AWS Serverless Implementation
We recently had experience implementing an event-driven framework for integration with disparate producers and consumers. The urgency of the integration required us to leverage AWS managed services to implement this use case.
Requirement
Events were being streamed into the system through objects being dropped into an AWS S3 bucket. We had started with a single consumer of this event. When an object gets dropped into the bucket the notification needed to trigger a workflow that would consume the objects from the bucket and publish it to an application exposed as a REST API.
Implementation
- Configured the create object Events in the S3 bucket to be published into an SQS queue
- The SQS queue is consumed by a lambda function that
- consumes the events,
- parses the objects
- transforms into a JSON format
- bundles multiple such JSON objects into bulk requests
- publishes it through the REST API
This worked as intended and we could leverage the AWS integration to handle the resilience and failure scenarios without much code to handle cross-cutting concerns.
The next challenge arrived with an additional consumer of this event arriving into the mix.
Alternate Implementation
- S3 Create Object events were configured to be published into an SNS Topic in place of the SQS queue
- The 2 consumers of the events have individual SQS queues configured
- The Queues subscribed to the SNS Topic created above
- The Queues had separate lambda functions that performed the necessary
- Transformation
- Processing
- REST invocations independently.
The takeaway from this implementation is the agility with which we could develop the solution and also have the ability to adapt it to the new scenario at a faster pace.
The adaptation also was done with the configuration on the AWS console without any custom code implementation, minimizing the effort involved in developing and testing the custom implementations that would have raised from such changes in requirements.
