Discover all tags used in all AWS resources with Golang and AWS GO SDK

As your AWS infrastructure grows, it can become increasingly difficult to keep track of all the resources you’ve created and their associated tags. However, having accurate and comprehensive resource tagging is important for many reasons, such as cost allocation, resource organization, and compliance auditing.

In this blog post, we’ll explore a solution that uses the AWS SDK to fetch all resource tags across all AWS regions and store them in a SQLite database. This solution can be run using multiple AWS credentials, allowing you to discover all tags for all resources across your entire AWS account.

Solution Overview

The solution consists of two main components: fetching the resource tags using the AWS SDK and storing the tags in a SQLite database. Let’s take a closer look at each component.

The final results would look like -

Resource ARN	Tag Key	Tag Value
arn:aws:ec2:us-east-2:xxxx:vpc-flow-log/fl-xxxxx	TagKey1	TagValue1
arn:aws:ec2:us-east-2:xxxx:s3/fl-xxxxx	TagKey1	TagValue2
arn:aws:ec2:eu-west-2:xxxx:ec2/fl-xxxx	TagKey4	TagValue5
arn:aws:ec2:us-east-2:xxx:rds/fl-xxxxx	TagKey6	TagValue8

Fetching Resource Tags

We’ll use the AWS SDK’s Resource Groups Tagging API to fetch the resource tags. Specifically, we’ll use the GetResources API call to list all resources in the account that have the specified tags. We’ll also use the PaginationToken response field to iterate through all the pages of resources.

To fetch all resource tags across all AWS regions, we’ll loop through a list of region codes and run the GetResources call for each region. We’ll also use multiple AWS credentials, allowing us to discover all tags for all resources across our entire AWS account.

Storing Resource Tags

We’ll store the resource tags in a SQLite database. Specifically, we’ll create a new resources table that has columns for the resource ARN, tag key, and tag value. We’ll use the Go SQLite library to create and interact with the database.

For each resource, we’ll loop through all the associated tags and insert them into the resources table. We’ll also keep track of the seen resource ARNs to avoid duplicates.

Code Explanation

Now that we have an overview of the solution, let’s take a closer look at the code.

First, we define a Cred struct to store our AWS credentials:

type Cred struct {
	Name            string
	AccessKeyID     string
	SecretAccessKey string
	SessionToken    string
}

We also define a list of region codes:

var regionCodes = []string{
	"us-east-2",
	"us-east-1",
	// ...
}

Next, we define a getAWSResourceTags function that fetches the resource tags for a given region:

func getAWSResourceTags(creds *credentials.Credentials, region string, seenARNs map[string]bool) error {
	sess, err := session.NewSession(&aws.Config{
		Credentials: creds,
		Region:      aws.String(region),
	})
	if err != nil {
		return err
	}

	// Create a new resourcegroupstaggingapi client
	client := resourcegroupstaggingapi.New(sess)

	// Filter resources by the "tribe" or "squad" tags
	input := &resourcegroupstaggingapi.GetResourcesInput{}

	db, err := prepareStorage()
	if err != nil {
		fmt.Printf("failed to setup storage. Error: %s\n", err.Error())
		return err
	}

	defer db.Close()

	tx, err := startDBTransaction(db)
	if err != nil {
		fmt.Printf("Error: %s\n", err.Error())
		return err
	}

	var stmt *sql.Stmt
	// List all resources in the account that have the specified tags
	for {
		output, err := client.GetResources(input)
		if err != nil {
			fmt.Println("Error listing resources:", err)
			return err
		}

		for _, resource := range output.ResourceTagMappingList {
			fmt.Printf("======\n ARN: %s\n", *resource.ResourceARN)
			arn := *resource.ResourceARN
			if !seenARNs[arn] {
				for _, tag := range resource.Tags {
					fmt.Printf("Tag: %s - %s\n\n", *tag.Key, *tag.Value)
					stmt, err = saveTagsAndResourceARN(tx, arn, tag)
					if err != nil {
						return err
					}
				}
				seenARNs[arn] = true
			}
			fmt.Printf("===== \n")
		}

		// Check if there are more pages to fetch
		if output.PaginationToken == nil || *output.PaginationToken == "" {
			break
		}

		// Set the NextToken to fetch the next page
		input.PaginationToken = output.PaginationToken
	}

	err = closeDBTransaction(tx, stmt)
	if err != nil {
		fmt.Printf("Error: %s\n", err.Error())
		return err
	}
	return nil
}

In this post, we demonstrated how to use the AWS SDK for Go to fetch all the tags for all resources in all AWS regions using multiple AWS accounts. We also showed how to store the data in a SQLite database for easy querying and analysis.

This solution can be useful for organizations that need to keep track of all their AWS resources and their associated tags for compliance or cost optimization purposes. It can also be helpful for developers who want to quickly discover all the tags for their AWS resources to debug issues or optimize their applications.

We hope this blog post was helpful to you. If you have any questions or feedback, please leave a comment below. Thank you for reading!

Full Source Code

package main

import (
	"database/sql"
	"fmt"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/resourcegroupstaggingapi"
	_ "github.com/mattn/go-sqlite3"
)

const (
	sqlite3DbPath = "./resources.db"
)

type Cred struct {
	Name            string
	AccessKeyID     string
	SecretAccessKey string
	SessionToken    string
}

func main() {

	var regionCodes = []string{
		"us-east-2",
		"us-east-1",
		"us-west-1",
		"us-west-2",
		"af-south-1",
		"ap-east-1",
		"ap-south-2",
		"ap-southeast-3",
		"ap-southeast-4",
		"ap-south-1",
		"ap-northeast-3",
		"ap-northeast-2",
		"ap-southeast-1",
		"ap-southeast-2",
		"ap-northeast-1",
		"ca-central-1",
		"eu-central-1",
		"eu-west-1",
		"eu-west-2",
		"eu-south-1",
		"eu-west-3",
		"eu-south-2",
		"eu-north-1",
		"eu-central-2",
		"me-south-1",
		"me-central-1",
		"sa-east-1",
		"us-gov-east-1",
		"us-gov-west-1",
	}

	cs := []Cred{
		{
			Name:            "Credentials_Identifier_If_You_want_to_use_multiple_credentials",
			AccessKeyID:     "AWS_ACCESS_KEY_ID",
			SecretAccessKey: "AWS_SECRET_ACCESS-KEY",
			SessionToken:    "AWS_SESSION_TOKEN",
		},
	}
	// Keep track of seen resource ARNs to avoid duplicates
	seenARNs := make(map[string]bool)

	for _, c := range cs {
		creds := credentials.NewStaticCredentials(c.AccessKeyID, c.SecretAccessKey, c.SessionToken)
		for _, r := range regionCodes {
			fmt.Printf("Creds: %s, Region: %s\n", c.Name, r)
			err := getAWSResourceTags(creds, r, seenARNs)
			if err != nil {
				fmt.Printf("Region: %s, Error: %s", r, err.Error())
				continue
			}
		}
	}

	fmt.Println("Finished")
}

func getAWSResourceTags(creds *credentials.Credentials, region string, seenARNs map[string]bool) error {
	sess, err := session.NewSession(&aws.Config{
		Credentials: creds,
		Region:      aws.String(region),
	})
	if err != nil {
		return err
	}

	// Create a new resourcegroupstaggingapi client
	client := resourcegroupstaggingapi.New(sess)

	// Filter resources by the "tribe" or "squad" tags
	input := &resourcegroupstaggingapi.GetResourcesInput{}

	db, err := prepareStorage()
	if err != nil {
		fmt.Printf("failed to setup storage. Error: %s\n", err.Error())
		return err
	}

	defer db.Close()

	tx, err := startDBTransaction(db)
	if err != nil {
		fmt.Printf("Error: %s\n", err.Error())
		return err
	}

	var stmt *sql.Stmt
	// List all resources in the account that have the specified tags
	for {
		output, err := client.GetResources(input)
		if err != nil {
			fmt.Println("Error listing resources:", err)
			return err
		}

		for _, resource := range output.ResourceTagMappingList {
			fmt.Printf("======\n ARN: %s\n", *resource.ResourceARN)
			arn := *resource.ResourceARN
			if !seenARNs[arn] {
				for _, tag := range resource.Tags {
					fmt.Printf("Tag: %s - %s\n\n", *tag.Key, *tag.Value)
					stmt, err = saveTagsAndResourceARN(tx, arn, tag)
					if err != nil {
						return err
					}
				}
				seenARNs[arn] = true
			}
			fmt.Printf("===== \n")
		}

		// Check if there are more pages to fetch
		if output.PaginationToken == nil || *output.PaginationToken == "" {
			break
		}

		// Set the NextToken to fetch the next page
		input.PaginationToken = output.PaginationToken
	}

	err = closeDBTransaction(tx, stmt)
	if err != nil {
		fmt.Printf("Error: %s\n", err.Error())
		return err
	}
	return nil
}

func closeDBTransaction(tx *sql.Tx, stmt *sql.Stmt) error {
	err := stmt.Close()
	if err != nil {
		fmt.Println("Error closing statement:", err)
		return fmt.Errorf("error closing statement. Error: %w", err)
	}
	err = tx.Commit()
	if err != nil {
		return fmt.Errorf("error committing transaction. Error: %w", err)
	}
	return nil
}

func saveTagsAndResourceARN(tx *sql.Tx, arn string, tag *resourcegroupstaggingapi.Tag) (*sql.Stmt, error) {
	fmt.Printf("  %s: %s\n", *tag.Key, *tag.Value)
	// Print the tags for each resource and save them to the database
	stmt, err := tx.Prepare("INSERT INTO resources(arn, key, value) values(?, ?, ?)")
	if err != nil {
		return nil, fmt.Errorf("error preparing statement. Error: %w", err)
	}
	_, err = stmt.Exec(arn, *tag.Key, *tag.Value)
	if err != nil {
		return nil, fmt.Errorf("error executing statement. Error: %w", err)
	}

	return stmt, nil
}

func startDBTransaction(db *sql.DB) (*sql.Tx, error) {
	tx, err := db.Begin()
	if err != nil {
		return nil, fmt.Errorf("error beginning transaction: %w", err)
	}

	return tx, nil
}

func prepareStorage() (*sql.DB, error) {
	// Open a new SQLite database file
	db, err := sql.Open("sqlite3", sqlite3DbPath)
	if err != nil {
		return nil, fmt.Errorf("error creating database connection. Error: %w", err)
	}

	// Create a new resources table if it doesn't already exist
	_, err = db.Exec(`CREATE TABLE IF NOT EXISTS resources (id INTEGER PRIMARY KEY, arn TEXT, key TEXT, value TEXT)`)
	if err != nil {
		return nil, fmt.Errorf("error creating initial table. Error: %w", err)
	}

	return db, nil
}

Just run go run . and it will start fetching all the resource ARN and it’s associated tags and store in the sqlite db. Later, you can use any database client to interact with that database to run query and analyze results.

Analyzing Resource Tags

Once you’ve gathered all your AWS resource tags into a database, you can start running some queries to analyze the data. Here are a few example queries you can run to get started:

Count the Number of Resources with Each Tag Key

SELECT key, COUNT(DISTINCT arn) AS count
FROM resources
GROUP BY key
ORDER BY count DESC;

This query will count the number of resources that have each unique tag key, and return the results sorted in descending order by count.

Find Resources with a Specific Tag Value

SELECT arn
FROM resources
WHERE key = 'Environment' AND value = 'Production';

This query will return a list of all resources with a tag key of “Environment” and a tag value of “Production”.

Find Resources Without Any Tags

SELECT arn
FROM resources
WHERE arn NOT IN (SELECT DISTINCT arn FROM resources WHERE key IS NOT NULL);

This query will return a list of all resources that do not have any tags associated with them.

Find Resource Types (Product) from AWS ARN

SELECT s.product AS product, COUNT(*) AS total
FROM (SELECT SUBSTR(REPLACE(arn, 'arn:aws:', ''), 1, INSTR(REPLACE(arn, 'arn:aws:', ''), ':') - 1) AS product
      FROM resources) AS s
GROUP BY s.product
ORDER BY COUNT(*) DESC

It will give you all the products you are using in AWS.

Resource Type	Number of Resources
rds	290
ec2	38
ecr	21
elasticache	7

These are just a few examples of the types of queries you can run to analyze your AWS resource tags. With your tags stored in a database, the possibilities are endless.

Solution Overview#

Fetching Resource Tags#

Storing Resource Tags#

Code Explanation#

Full Source Code#

Analyzing Resource Tags#