The Good, The Bad, and How To Do It With Terraform

AWS Transfer Family is a fully managed file transfer service enabling secure and seamless data migration, storage, and sharing between on-premises and cloud storage. The service supports popular file transfer protocols such as SFTP, FTPS, and FTP. Today, I’ll explore the service’s essential features, use cases, benefits, and pitfalls. I’ll also explain a basic example of implementing the Transfer Family service in Terraform.

Key Features

The service is good when using SFTP, FTPS, AS2, or plain FTP. Most people would default to running one on an EC2, but why bother needing to worry about OS and software patching? The managed service is also much more scalable and flexible than an EC2 server would be.

  1. Protocol Support: AWS Transfer Family supports three widely-used file transfer protocols — SFTP (Secure File Transfer Protocol), FTPS (File Transfer Protocol Secure), AS2 (Applicability Statement 2), and FTP (File Transfer Protocol). By supporting these protocols, the service ensures compatibility with existing file transfer workflows, making migration to the cloud straightforward.
  2. Integration with AWS Services: The Transfer Family integrates with other AWS services like Amazon S3 and Amazon EFS for file storage. Additionally, you can use it alongside AWS Identity and Access Management (IAM) for authentication and authorization, AWS Key Management Service (KMS) for data encryption, and Amazon CloudWatch for monitoring and logging.
  3. Custom Domain and Identity Providers: AWS Transfer Family allows you to use custom domain names for your file transfer endpoints, enabling a seamless experience for your users. Moreover, it can integrate with existing identity providers (IdP) using AWS Lambda, allowing you to authenticate users through your existing systems.
  4. High Availability and Scalability: The service provides high availability through AWS’s global infrastructure and automatically scales with your workload, ensuring consistent performance even during peak usage periods.
  5. Security and Compliance: AWS Transfer Family complies with industry standards and certifications, such as HIPAA, PCI DSS, and FedRAMP, ensuring your data remains secure and adheres to regulatory requirements. It also supports encryption at rest and in transit, providing additional layers of security for your sensitive data.

Use Cases

The service has plenty of use cases, but I wanted to highlight two.

Data Migration: AWS Transfer Family simplifies migrating large volumes of data from on-premises storage to the AWS cloud. Using regular file transfer protocols, you can quickly transfer data to Amazon S3 or Amazon EFS without custom code or infrastructure.

Data Sharing and Collaboration: Businesses often need to share files with external partners, vendors, or customers. With the Transfer Family, you can provide secure, controlled access to your files stored in Amazon S3 or EFS, enabling seamless collaboration. This service is a popular choice in the banking and finance industries.

Benefits

  1. Simplified File Transfer: The AWS Transfer Family streamlines the process of transferring files to and from the AWS cloud by supporting widely-used file transfer protocols. This compatibility with existing workflows makes it easier for organizations to adopt and leverage cloud storage without significant changes to their current processes.
  2. Enhanced Security: By integrating with AWS’s security services, such as IAM and KMS, the Transfer Family ensures that your data remains secure throughout its lifecycle. Additionally, the service complies with various industry standards, providing peace of mind when meeting regulatory requirements.
  3. Seamless Integration: AWS Transfer Family’s tight integration with other AWS services, such as Amazon S3 and Amazon EFS, enables you to store, process, and analyze your data in a unified environment. This seamless integration simplifies data management, reduces operational overheads, and allows you to unlock valuable insights from your data more efficiently.
  4. Improved Collaboration: The service enables secure, controlled access to your data stored in Amazon S3 or EFS, facilitating collaboration with external partners, vendors, or customers. By providing a familiar interface and support for standard file transfer protocols, AWS Transfer Family eliminates the need for complex, custom-built solutions for data sharing.
  5. Scalability and Availability: Leveraging AWS’s global infrastructure, the Transfer Family automatically scales with your workload and ensures high availability. This capability guarantees consistent performance even during peak usage periods and allows you to focus on your core business rather than managing infrastructure.

Downsides

  1. Cost: While AWS Transfer Family simplifies file transfers, monitoring costs is essential, as pricing is based on usage. Be aware of data transfer, storage, and request costs to avoid unexpected expenses.
  2. Data Transfer Performance: Transferring large volumes of data over the internet can be slow and unreliable. Consider using AWS Direct Connect for a dedicated network connection or AWS Snowball for offline data transfer to mitigate this.
  3. Access Control and Security: While the AWS Transfer Family provides built-in security features, it’s crucial to configure access controls and encryption properly. These configurations can prove challenging when combining identity policies (IAM) and resource policies (KMS and S3) in various configurations. Ensure that appropriate IAM policies are in place and data encryption is enabled in transit and at rest.

Terraform Example

The Terraform snippet below can be found in its full context in the repo: mencarellic/terraform-aws-transfer-family.

The first thing to do is create an IAM role and policy for the service and users of the service. There isn’t anything special here. You allow Transfer Family to assume the role and will enable the role access to the bucket that was created.

resource "aws_iam_role" "transfer_role" {
 name = "example-transfer-role"

 assume_role_policy = jsonencode({
   Version = "2012-10-17"
   Statement = [
    {
       Action = "sts:AssumeRole"
       Effect = "Allow"
       Principal = {
         Service = "transfer.amazonaws.com"
      }
    }
  ]
})
}

resource "aws_iam_role_policy" "transfer_policy" {
 name = "example-transfer-policy"
 role = aws_iam_role.transfer_role.id

 policy = jsonencode({
   Version = "2012-10-17"
   Statement = [
    {
       Action = [
         "s3:ListBucket",
         "s3:GetBucketLocation",
      ]
       Effect   = "Allow"
       Resource = aws_s3_bucket.transfer_bucket.arn
    },
    {
       Action = [
         "s3:PutObject",
         "s3:GetObject",
         "s3:DeleteObject",
         "s3:AbortMultipartUpload",
         "s3:ListMultipartUploadParts",
      ]
       Effect   = "Allow"
       Resource = "${aws_s3_bucket.transfer_bucket.arn}/*"
    }
  ]
})
}

Next, you’ll create the actual server and user. Check out the latest Terraform documentation for the resource for more details. It’s pretty self explanatory though. The only one I usually need to look up is the seurity policy which is similar to the ELB security policy in that it’s a set of cryptographic policies that are allow. See the AWS documentation for more details on available policies.

resource "aws_transfer_server" "sftp_server" {
  identity_provider_type = "SERVICE_MANAGED"
  logging_role           = aws_iam_role.transfer_role.arn
  protocols              = ["SFTP"]
  security_policy_name   = "TransferSecurityPolicy-2022-03"

}

resource "aws_transfer_user" "sftp_user" {
  server_id = aws_transfer_server.sftp_server.id
  user_name = "example-user"
  role      = aws_iam_role.transfer_role.arn

  home_directory_type = "LOGICAL"

  home_directory_mappings {
    entry  = "/"
    target = "/${aws_s3_bucket.transfer_bucket.id}/$${Transfer:UserName}"
  }
}

After that, you must attach a public SSH key to the user created. You can do that in the AWS Console, via the CLI, or with the aws_transfer_ssh_key Terraform resource. For my example, I had an SSH key already generated, so I just added it via the GUI.

Once Terraform applies, you can run the traditional SFTP commands like GET and PUT.

Conclusion

Transfer Family is a powerful, fully managed file transfer service that simplifies data migration, storage, and sharing between on-premises and cloud environments. By understanding the key features, use cases, challenges, and benefits of the AWS Transfer Family, you can leverage this service to its full potential and maximize the value of their data in the AWS cloud. Supporting popular file transfer protocols and integrating seamlessly with other AWS services provides a comprehensive solution for managing your data securely and efficiently. However, it’s essential to be aware of potential pitfalls, such as cost management and data transfer performance, and take appropriate measures to mitigate them.

References