Terraform 12 Tutorial - Creating AWS S3 bucket / SQS queue resources and notifying bucket event to queue
bogotobogo.com site search:
Prerequisites
$ cat ~/.aws/config [default] region = us-west-1 $ cat ~/.aws/credentials [default] aws_access_key_id = AKI...6 aws_secret_access_key = wh...CUe $ terraform version Terraform v0.12.28 + provider.aws v3.16.0 + provider.random v3.0.0
S3 and SQS queue
Here is the file:
s3_sqs.tf:
provider "aws" {
profile = "default"
region = "us-west-1"
}
resource "random_string" "random" {
length = 6
special = false
upper = false
}
resource "aws_s3_bucket" "mys3bucket" {
bucket = "mys3bucket-${random_string.random.result}"
}
resource "aws_sqs_queue" "q" {
name = "s3-event-queue"
policy = <<POLICY
{
"Version": "2012-10-17",
"Id": "sqspolicy",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:*:*:s3-event-queue",
"Condition": {
"ArnEquals": { "aws:SourceArn": "${aws_s3_bucket.mys3bucket.arn}" }
}
}
]
}
POLICY
}
resource "aws_s3_bucket_notification" "bucket_notification" {
bucket = aws_s3_bucket.mys3bucket.id
queue {
queue_arn = aws_sqs_queue.q.arn
events = ["s3:ObjectCreated:*"]
}
}
Let's run the file:
$ terraform init
$ terraform plan
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_s3_bucket.mys3bucket will be created
+ resource "aws_s3_bucket" "mys3bucket" {
+ acceleration_status = (known after apply)
+ acl = "private"
+ arn = (known after apply)
+ bucket = (known after apply)
+ bucket_domain_name = (known after apply)
+ bucket_regional_domain_name = (known after apply)
+ force_destroy = false
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ region = (known after apply)
+ request_payer = (known after apply)
+ website_domain = (known after apply)
+ website_endpoint = (known after apply)
+ versioning {
+ enabled = (known after apply)
+ mfa_delete = (known after apply)
}
}
# aws_s3_bucket_notification.bucket_notification will be created
+ resource "aws_s3_bucket_notification" "bucket_notification" {
+ bucket = (known after apply)
+ id = (known after apply)
+ queue {
+ events = [
+ "s3:ObjectCreated:*",
+ "s3:ObjectRemoved:*",
]
+ id = (known after apply)
+ queue_arn = (known after apply)
}
}
# aws_sqs_queue.q will be created
+ resource "aws_sqs_queue" "q" {
+ arn = (known after apply)
+ content_based_deduplication = false
+ delay_seconds = 0
+ fifo_queue = false
+ id = (known after apply)
+ kms_data_key_reuse_period_seconds = (known after apply)
+ max_message_size = 262144
+ message_retention_seconds = 345600
+ name = "s3-event-queue"
+ policy = (known after apply)
+ receive_wait_time_seconds = 0
+ visibility_timeout_seconds = 30
}
# random_string.random will be created
+ resource "random_string" "random" {
+ id = (known after apply)
+ length = 6
+ lower = true
+ min_lower = 0
+ min_numeric = 0
+ min_special = 0
+ min_upper = 0
+ number = true
+ result = (known after apply)
+ special = false
+ upper = false
}
Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
random_string.random: Creating...
random_string.random: Creation complete after 0s [id=9xc559]
aws_s3_bucket.mys3bucket: Creating...
aws_s3_bucket.mys3bucket: Creation complete after 6s [id=mys3bucket-9xc559]
aws_sqs_queue.q: Creating...
aws_sqs_queue.q: Still creating... [10s elapsed]
aws_sqs_queue.q: Still creating... [20s elapsed]
aws_sqs_queue.q: Creation complete after 26s [id=https://sqs.us-west-1.amazonaws.com/526262051452/s3-event-queue]
aws_s3_bucket_notification.bucket_notification: Creating...
aws_s3_bucket_notification.bucket_notification: Creation complete after 1s [id=mys3bucket-9xc559]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
We can check if the S3 bucket and sqs queue have been created:
$ aws s3 ls
2020-11-20 22:26:49 mys3bucket-9xc559
$ aws sqs list-queues --region=us-west-1
{
"QueueUrls": [
"https://sqs.us-west-1.amazonaws.com/526262051452/s3-event-queue"
]
}
Upload a file to the the bucket and check if the file is there:
$ aws s3 cp myfile s3://mys3bucket-9xc559 upload: ./myfile to s3://mys3bucket-9xc559/myfile $ aws s3 ls s3://mys3bucket-9xc559 2020-11-20 23:44:17 6 myfile
Fetch and read the message:
$ aws sqs --region us-west-1 receive-message --queue-url https://sqs.us-west-1.amazonaws.com/526262051452/s3-event-queue
{
"Messages": [
{
"MessageId": "0c5a2121-f5f7-4984-85f6-dd4d7ea0a1dd",
"ReceiptHandle": "AQEBNYBX8pL+4Fke2MIrH20tmUSYAXUtrdAuFrACcG8lrn/h5JmNbFgsLlfwIVTDvllgshUaj9fedq8rJXqMw08spvrBs+68sZu+DJz/zhksxuqAzpTNUHLOhlQUXCjEJdm6KE0AptLw9L0L693geekUBpXGiHN/yH0XRoa2D/dmZ4ZPN+A7f20TzuGmaoNq/iQISzDMFp/6AdoYZIHTnEAaInCNRWfmDGWbKdMY5fPZh4YUWuPz6cvuCNFZpJAbmsfvGT6mXXezoApwAB0OdguKLNFgRYk+EBYVSSMch2xwDvBduW7HWqDkHswxte1Vof16NQPtnEylUMRl2JZxQLbOEHE+TtTh/m300x9Sj/ES+1WWKcJho2+KYi9IdFphSdkfJfPCe9PAik3gwXhPgTm8/w==",
"MD5OfBody": "d890297da36b74660f261dcf9224fad4",
"Body": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-west-1\",\"eventTime\":\"2020-11-21T07:44:12.905Z\",
\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:AIDAJIX6NM5BAIKEDHN3W\"},\"requestParameters\":{\"sourceIPAddress\":\"73.70.219.237\"},
\"responseElements\":{\"x-amz-request-id\":\"A62E0EF808370AD7\",\"x-amz-id-2\":\"yU/scs9ylw31g2M2LX+Oy1IIm5kqPKVJqBFG2DoqgOVHV0bS7fhnfegCLxgqpqfqY0is29UvIc53WpZKBJ6jSCdmegNFEx9k\"},
\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"tf-s3-queue-20201121062721350500000001\",
\"bucket\":{\"name\":\"mys3bucket-9xc559\",\"ownerIdentity\":{\"principalId\":\"A2CIWRO9QISUM0\"},
\"arn\":\"arn:aws:s3:::mys3bucket-9xc559\"},
\"object\":{\"key\":\"myfile\",\"size\":6,\"eTag\":\"d577273ff885c3f84dadb8578bb41399\",\"sequencer\":\"005FB8C550D449FD18\"}}}]}"
}
]
}
We can see the uploading myfile event is in the sqs queue!
Clean up the S3 and SQS queue
$ aws s3 rm s3://mys3bucket-9xc559/myfile
delete: s3://mys3bucket-9xc559/myfile
$ terraform destroy
random_string.random: Refreshing state... [id=9xc559]
aws_s3_bucket.mys3bucket: Refreshing state... [id=mys3bucket-9xc559]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
# aws_s3_bucket.mys3bucket will be destroyed
- resource "aws_s3_bucket" "mys3bucket" {
- acl = "private" -> null
- arn = "arn:aws:s3:::mys3bucket-9xc559" -> null
- bucket = "mys3bucket-9xc559" -> null
- bucket_domain_name = "mys3bucket-9xc559.s3.amazonaws.com" -> null
- bucket_regional_domain_name = "mys3bucket-9xc559.s3.us-west-1.amazonaws.com" -> null
- force_destroy = false -> null
- hosted_zone_id = "Z2F56UZL2M1ACD" -> null
- id = "mys3bucket-9xc559" -> null
- region = "us-west-1" -> null
- request_payer = "BucketOwner" -> null
- tags = {} -> null
- versioning {
- enabled = false -> null
- mfa_delete = false -> null
}
}
# random_string.random will be destroyed
- resource "random_string" "random" {
- id = "9xc559" -> null
- length = 6 -> null
- lower = true -> null
- min_lower = 0 -> null
- min_numeric = 0 -> null
- min_special = 0 -> null
- min_upper = 0 -> null
- number = true -> null
- result = "9xc559" -> null
- special = false -> null
- upper = false -> null
}
Plan: 0 to add, 0 to change, 2 to destroy.
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_s3_bucket.mys3bucket: Destroying... [id=mys3bucket-9xc559]
aws_s3_bucket.mys3bucket: Destruction complete after 1s
random_string.random: Destroying... [id=9xc559]
random_string.random: Destruction complete after 0s
Destroy complete! Resources: 2 destroyed.
Terraform
Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization
My YouTube channel