Data Ingestion using Database Migration Service(DMS) and Lambda
Last updated
Was this helpful?
Last updated
Was this helpful?
The is a managed service to migrate data into AWS. It can replicate data from operational databases and data warehouses (on premises or AWS) to a variety of targets, including S3 datalakes. In this architecture, DMS is used to capture changed records from relational databases on RDS or EC2 and write them into S3. , a serverless compute service, is used to transform and partition datasets based on their arrival time in S3 for better query performance.
Create a Relational databases on EC2 or RDS within a VPC.
Create a Staging S3 location to store changes captured by DMS.
using the DMS API's or console
for the Replication Instance.
which has read access on the staging S3 bucket and write access on target datalake location.
to trigger execution with s3:ObjectCreated:*
requests to the staging S3 bucket. The function writes the same objects to the target datalake location on S3 with partitions based on the metadata attribute of S3 objects.
to migrate data from your source system to target location.
The DMS Replication Instance will then connect to the source via , and write to the S3 staging location. AWS Lambda will receive the PutObject events, and use the to reorganise the data into your datalake.