AWS DataSync – The data migration hub in your hybrid environment
Migrating large amounts of data from your traditional file server environments to AWS can be a challenge. Especially if the transfer has to be completed in a short time frame.
One possible solution can be the use of AWS DataSync. DataSync belongs to the family of existing AWS migration tools like Server Migration Service (SMS) and Database Migration Service (DMS).
The migration process is described via a so-called „tasks” within the DataSync service and each task defines a source and the corresponding target.
| DataSync Sources | DataSync Targets |
|---|---|
| NFS Share | S3 |
| SMB Share | EFS |
| S3 | FSx for Windows |
| EFS |
- Up to 100 tasks can be started simultaneously per account within a region
- The per-task limit for transferred files is set to a maximum of 50 million files
- AWS DataSync is activated either by downloading and starting a VM in the physical data center or by using an Amazon Machine Image (AMI) in the AWS environment.
- In both cases, we speak of the AWS DataSync Agent. The following hypervisor environments are supported:
- VMware ESXi Hypervisor version 6.0 or 6.5 or 6.7
- AWS EC2 Hypervisor
The following section presents two scenarios in which AWS DataSync can successfully deployed:
1 Migrating file server data from an on-premise data center to AWS
2 Data migration within an AWS Region or migration to another AWS Region
Implementation within a customer project
Initial situation
- 4 TB data (PDF and Word files) on SMB fileshare in the Data Center
- Classic Windows Server application layer
- Application needs mount volumes for data access
- Application reads 99% of data in Object Mode
- Migration of the data must be completed within 4 weeks
- Newly generated and modified data in the source directory must be available after Go Live
Implementation and target scenario
- Use of AWS DataSync
- Use of multiple tasks simultaneously for initial and incremental migration to S3 Buckets
- 3rd party solution “CloudBerry” for native mounting of the respective S3 buckets for the Windows applications
- On average 1000 read accesses per minute per Windows node