Question: How do I troubleshoot and reduce increased latency from CloudFront?

I'm seeing increased latency in responses from Amazon CloudFront. How can I identify the cause and reduce latency?

Answer: "I'm seeing increased latency in responses from Amazon CloudFront. How can I identify the cause and reduce latency?Short descriptionTo troubleshoot latency from CloudFront, first identify which of the following events are contributing to the latency:The time taken for requests to go between the Client and the CloudFront edge locations. This includes the Domain Name System (DNS) lookup process and TCP and SSL/TLS negotiations.The time taken for requests to go between CloudFront and the origin. This includes the origin DNS lookup process, TCP and SSL/TLS negotiations with the origin, and time taken by the origin to process and respond with an object.Then, follow the troubleshooting steps for the events that are causing the most latency.ResolutionIdentify events causing latency from CloudFront:To identify which events are causing latency from CloudFront, do one of the following:Run the following cURL command:curl -w "DNS_resolution: %{time_namelookup}| TCP_negotiation_time: %{time_connect}| SSL_negotiation_time: %{time_appconnect}| TTFB: %{time_starttransfer}| Total time: %{time_total} \n" -o /dev/null -vsL https://www.example.comNote: Replace example.com with your CloudFront domain name or alternate domain name (CNAME) and URL path.Check how long each stage of a network request takes in the developer tools of your web browser. For example, if you're using Mozilla Firefox, the Timing tab provides this information.Based on the time-taken for each event or request, see the related Resolution section in this article.If you observed latency in the past, then check the fields time-taken and time-to-first-byte in the CloudFront access logs. CloudFront access logs don't log the time taken by the client for the DNS lookup process and TCP and SSL/TLS negotiationsReducing latency in DNS resolutionIncrease DNS caching time in the client-side DNS.Increase the Time to Live (TTL) of the cache in the local DNS server.Increase the TTL on the DNS record in your DNS registrar/provider.If the resolver DNS server from your internet service provider is causing latency, then consider using public DNS servers.Reducing latency in TCP and SSL/TLS - Negotiation timeCheck your local network bandwidth and internet bandwidth.Check if there are any network disruption in your internet server provider or router.Optimize your local network performance through your internet service provider or network routes.Confirm you're using the correct DNS resolver that allows your web browser to find the closest and correct POP location.To improve HTTPS site performance, keep your certification chain short.Latency can be caused by a firewall, proxy, or local router. To determine which of these is causing latency, run the following MTR command from your system. See Diagnosing Network Issues with MTR for more information.mtr -rw example.com --no-dnsNote: Replaceexample.com with your domain name.Reducing latency in Time Taken for First Byte (TTFB) and Total Time Taken (TTL)If CloudFront returns "X-Cache:Hit from cloudfront"CloudFront returns "X-Cache:Hit from cloudfront" when requests are served from the closest edge location. To reduce latency:Turn on CloudFront automatic compression to compress files and increase download speed.Use local or browser caching to reduce requests to CloudFront. Specify cache-control header on the files to instruct web browsers to keep the website content in the browser's memory or the local disk for a certain amount of time. For more information on cache-control headers, see Specifying the amount of time that CloudFront caches objects.If CloudFront returns "X-Cache:Miss from cloudfront"If CloudFront returns "X-Cache:Miss from cloudfront" when the request is sent to the origin. To reduce latency:Reduce the Round trip time (RTT) between CloudFront edge location to your origin location. If a request from a CloudFront edge location goes to the nearest origin location, then the RTT is less. However, TTFB is affected if the request is from an edge location geographically distant from the origin. To optimize the RTT, replicate origin server in multiple regions that are geographically closer to your users. Then, set up the DNS of your origin domain name so that it routes the request to origin servers based on latency or geolocation. If you're using Amazon Route 53 as your DNS provider, then see Choosing a routing policy for more information.Turn on CloudFront automatic compression to compress files and reduce download speed. If your file format is not supported by CloudFront Automatic Compression, then pre-compress that file in your origin and serve it with Content-Encoding header.Check the latency from the origin to CloudFront by enabling the Origin latency metric. Note: Standard CloudWatch rates apply.Turn on CloudFront Origin Shield.Add Response headers policies with Server-Timing header feature turned on. This feature can help you understand the events that are contributing to the latency between CloudFront and the origin.Related informationHow do I reduce the latency of slow responses from CloudFront?How can I troubleshoot latency in my static website that's hosted on Amazon S3 and served through CloudFront?What information should I give AWS Support for diagnosing CloudFront web distribution latency issues?How do I determine what's causing request latency for my edge-optimized API endpoint in API Gateway?Follow"


Question: How can I configure cross-Region VPC interface endpoints for AWS services?

I want to configure cross-Region Amazon Virtual Private Cloud (Amazon VPC) endpoints to access AWS resources using a private link.

Answer: "I want to configure cross-Region Amazon Virtual Private Cloud (Amazon VPC) endpoints to access AWS resources using a private link.Short descriptionYou can deploy resources such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon VPC, and Amazon Relational Database Service (Amazon RDS) in different AWS Regions. This deployment aids in high availability of the resources and provides faster data access to users. You can also deploy VPC gateway endpoints to access AWS public resources, such as Amazon Simple Storage Service (Amazon S3), through a private link. However, you can access these VPC gateway endpoints only from the same Region.The following is an example scenario. You deploy an Amazon S3 gateway endpoint in the us-west-2 Region. You then have access to S3 buckets in us-west-2 through the gateway endpoint. Traffic to buckets in other Regions travels over the internet gateway of the VPC. If your source AWS resource is in a private subnet with a NAT gateway, then the traffic flows through the NAT gateway. Due to this traffic flow, the NAT gateway incurs higher charges than a VPC endpoint. Because there's no processing or hourly charges for using gateway VPC endpoints, Amazon S3 gateway endpoints are price efficient by default.Note: The following example uses Amazon S3 interface endpoints for cross-Region traffic because gateway endpoints don't support cross-Region access. Use the same setup for any VPC interface endpoint.ResolutionUse the following steps to create VPC peering between VPCs to access endpoints in a different Region.Note: For this example resolution, the following variables are used:VPC1(10.100.10.0/24) is in the us-east-1 Region.VPC1 has an Amazon S3 interface endpoint.VPC2(172.16.20.0/24) is in the us-east-2 Region.Users from the us-east-2 Region want to access the S3 bucket in us-east-1. They want to use the Amazon S3 interface endpoint in us-east-1.Configure VPC peering between VPC1 and VPC21.    Open the Amazon VPC console. Make sure that you are in the us-east-1 Region.2.    Choose VPC peering connections.3.    Choose Create peering connection.4.    Enter a Name for the peering connection.5.    For Select a local VPC to peer with, enter the VPC ID (in this example, the VPC ID for VPC1).6.    In Select another VPC to peer with, for Account, select the relevant option depending on the following:If this is a remote VPC that belongs to same AWS account, then choose My account.If this isn't a remote VPC that belongs to the same account, then choose Another account, and then enter the Account ID.7.    In Select another VPC to peer with, for Region, select Another Region. Then, enter the remote VPC ID that you want (in this example, the VPC ID for VPC2).8.    Choose Create peering connection. The peering connection status changes to pending acceptance.9.    Change the Region to us-east-2.10.   Open the Amazon VPC console, and then choose VPC peering connections.11.   Choose Actions, Accept request.Update the subnet route table and route table target1.    Add a route in the subnet route table for the us-east-1 endpoint for 172.16.20.0/24 (VPC2).2.    Add a route in the user's route table target in us-east-2 for 10.100.10.0/24 (VPC1) as a peering connection (pcx-xxxxxxxxxxxxxx).Access the S3 bucketAccess the S3 bucket using VPC endpoint FQDN from the remote VPC:aws s3 --region us-east-1 --endpoint-url https://bucket.vpce-xxxxxxxxxxx.s3.us-east-1.vpce.amazonaws.com ls s3://my-bucket/TroubleshootingMake sure that local and remote VPC subnets' route tables have routes that target each other as peer connections.The VPC endpoint permission policy must allow the remote VPC ID.Security groups applied to VPC endpoints must allow the remote VPC subnets.Follow"


Question: How do I troubleshoot issues with major version upgrades in Amazon RDS for PostgreSQL and Aurora for PostgreSQL?

My engine version upgrade for Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition is stuck or has failed.

Answer: "My engine version upgrade for Amazon Relational Database Service (Amazon RDS) for PostgreSQL or Amazon Aurora PostgreSQL-Compatible Edition is stuck or has failed.Short descriptionWhen Amazon RDS supports a new version of a database engine, you can upgrade your DB instances to the new version. You can perform a minor version grade or a major version upgrade for your DB instances.Minor version upgrades are used to patch security vulnerabilities and fix bugs. These upgrades usually don't add any new functionality, and don't change the internal storage format. They are always compatible with the earlier and later minor releases of the same major version. However, major version upgrades contain database changes that aren't backward-compatible with existing applications. These upgrades might change the internal format of system tables, data files, and data storage. Amazon RDS uses the PostgreSQL utility pg_upgrade to perform major version upgrades.During a major version upgrade of a PostgreSQL instance, Amazon RDS runs a precheck procedure. This procedure identifies any issues that might cause the upgrade to fail. It checks for potential incompatible conditions across all databases. If Amazon RDS identifies an issue during the precheck process, it creates a log event for the failed precheck. For more information on the precheck process for all databases, check the pg_upgrade_precheck.log upgrade log. Amazon RDS appends a timestamp to the file name. RDS events might also provide the reasons for upgrade failure. But for issues that are engine specific, you must check the database log files.For more information, see Viewing and listing database log files for RDS for PostgreSQL. Or, see Viewing and listing database log files for Aurora for PostgreSQL.During a major version upgrade, RDS completes these steps:Create a snapshot of the instance before the upgrade. This happens only if you set the backup retention period for your DB instance to a number greater than zero.Shut down the instance.Use the pg_upgrade utility to run the upgrade job on the instance.Create a snapshot of the instance after the upgrade.ResolutionAlthough Amazon RDS manages these upgrades, you might encounter the following issues during a version upgrade:Upgrade takes a longer time.Upgrade fails.Upgrade takes a long timePending maintenance activities: Any pending maintenance activities are automatically applied with engine version upgrades. This might include applying an operating system patch on your RDS instance. In this case, the operating system patch is applied first, and then the engine version is upgraded. So, performing operating system maintenance activities leads to an increase in the time taken to complete the upgrade.Also, if your RDS instance is in a Multi-AZ deployment, then operating system maintenance results in a failover. When you set up your instance in Multi-AZ, the backup for the instance is usually created on the secondary instance. In case of a failover, a backup is created on a new secondary instance after the upgrade. This backup on the new secondary instance might not be the latest backup. So a full backup might be triggered instead of an incremental backup. Creating a full backup can take a long time, especially if the database is very large.To avoid this issue, look for pending maintenance activities in the Pending maintenance section in your RDS console. For Aurora for PostgreSQL, see Viewing pending maintenance.Or, use the AWS Command Line Interface (AWS CLI) command describe-pending-maintenance-actions on your instance.aws rds describe-pending-maintenance-actions --resource-identifier example-arnNote: Complete these maintenance activities before performing the database engine version upgrades.No snapshot created before the upgrade: It's a best practice to create a snapshot of the RDS or Aurora for PostgreSQL cluster snapshot before performing the upgrade. If you already turned on backups for your instance, then a snapshot is created automatically as part of the upgrade process. Creating a snapshot before the upgrade reduces the time needed for the upgrade process to complete. This is because only an incremental backup is created during the upgrade process in this case.RDS for PostgreSQL Read replica upgrades: When you perform a major version upgrade of your primary DB instance, all the read replicas in the same Region are automatically upgraded. After the upgrade workflow starts, the read replicas wait for pg_upgrade to complete successfully on the primary DB instance. Then, the primary instance upgrade waits for the read replica upgrades to complete. You experience an outage until all the upgrades are complete. If the downtime window for the upgrade is limited, then you can promote or drop your replica instance. Then, recreate the read replicas after the upgrade is complete.To safely upgrade the DB instances that make up your cluster, Aurora for PostgreSQL uses the pg_upgrade utility. After the writer upgrade completes, each reader instance experiences a brief outage while it's upgraded to the new major version.Long-running transactions or high workload before the upgrade: Long-running transactions or high workload before the upgrade might increase the time taken to shut down the database and increase upgrade time.Run this query to identify long-running transactions:SQL>SELECT pid, datname, application_name, state, age(query_start, clock_timestamp()), usename, query FROM pg_stat_activity WHERE query NOT ILIKE '%pg_stat_activity%' AND usename!='rdsadmin' ORDER BY query_start desc;Insufficient compute capacity: The pg_upgrade utility can be compute-intensive. So, it's a best practice to perform a dry-run upgrade before upgrading your production databases. You can restore a snapshot of the production instance and perform a dry run with the same instance class as that of the production database.Upgrade failsUnsupported DB instance classes: The upgrade might fail if the instance class of your DB instance isn't compatible with the PostgreSQL version that you're upgrading to. Be sure to check the compatibility of the instance class with the engine version. For more information, review the supported DB engines for DB instance classes for RDS for PostgreSQL. Or, review the supported DB engines for DB instance classes for Aurora for PostgreSQL.Open prepared transactions: Prepared transactions that are open on the database might lead to upgrade failure. Be sure to commit or roll back all open prepared transactions before starting an upgrade.Run this query to check if there are open prepared transactions on your instance:SELECT count(*) FROM pg_catalog.pg_prepared_xacts;In this case, the error in the pg_upgrade.log file looks similar to this:------------------------------------------------------------------------Upgrade could not be run on Wed Apr 4 18:30:52 2018-------------------------------------------------------------------------The instance could not be upgraded from 9.6.11 to 10.6 for the following reasons.Please take appropriate action on databases that have usage incompatible with the requested major engine version upgrade and try the upgrade again.*There are uncommitted prepared transactions. Please commit or rollback all prepared transactions.*Unsupported data types: The upgrade fails with an error if you attempt to upgrade the database with unsupported data types, such as the following:regcollationregconfigregdictionaryregnamespaceregoperregoperatorregprocregprocedureNote: The data types regclass, regrole, and regtype are supported.The PostgreSQL upgrade utility pg_upgrade doesn't support upgrading databases that include table columns using the reg* OID-referencing system data types. Remove all uses of reg* data types, except for regclass, regrole, and regtype, before attempting an upgrade.Run this query to verify the usage of unsupported reg* data types:SELECT count(*) FROM pg_catalog.pg_class c, pg_catalog.pg_namespace n, pg_catalog.pg_attribute a  WHERE c.oid = a.attrelid      AND NOT a.attisdropped      AND a.atttypid IN ('pg_catalog.regproc'::pg_catalog.regtype,                         'pg_catalog.regprocedure'::pg_catalog.regtype,                         'pg_catalog.regoper'::pg_catalog.regtype,                         'pg_catalog.regoperator'::pg_catalog.regtype,                         'pg_catalog.regconfig'::pg_catalog.regtype,                         'pg_catalog.regdictionary'::pg_catalog.regtype)      AND c.relnamespace = n.oid      AND n.nspname NOT IN ('pg_catalog', 'information_schema');Logical replication slots: An upgrade can't occur if your instance has any logical replication slots. Logical replication slots are typically used for AWS Database Migration Service (AMS DMS) migration. They are also used for replicating tables from databases to data lakes, business intelligence tools, and other targets. Before upgrading, be sure that you know the purpose of logical replication slots that are in use and confirm that they can be deleted.If the logical replication slots are still being used, you must not delete them. In this case, you can't proceed with upgrade.The related error in the pg_upgrade log file looks similar to this example:"The instance could not be upgraded because one or more databases have logical replication slots. Please drop all logical replication slots and try again."If the logical replication slots aren't needed, run these queries to delete them:SELECT * FROM pg_replication_slots;SELECT pg_drop_replication_slot(slot_name);Storage issues: While the pg_upgrade script runs, the instance might run out of space. This causes the script to fail, and you see an error message similar to this:pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] could not execute query: ERROR: could not create file "base/12345/12345678": No space keyword" left  on deviceTo resolve this issue, be sure that the instance has sufficient free storage before starting the upgrade.Unknown data types: PostgreSQL versions 10 and later don't support unknown data types. If a PostgreSQL version 9.6 database uses the unknown data type, an upgrade to version 10 shows an error message like this:"The instance could not be upgraded because the 'unknown' data type is used in user tables. Please remove all usages of the 'unknown' data type and try again."This is a PostgreSQL limitation, and RDS automation doesn't modify columns using the unknown data type. You might need to modify these columns manually before the upgrade.Run this query to find columns in your database with unknown data type:SELECT DISTINCT data_type FROM information_schema.columns WHERE data_type ILIKE 'unknown';After identifying the columns, you can remove these columns or modify them to a supported data type.Read replica upgrade failure (RDS for PostgreSQL only): PostgreSQL instance has read replicas, then read replica upgrade failures might cause your primary instance upgrade to get stuck. Read replica upgrade failure might also result in failure of the primary instance upgrade. A failed read replica is placed in the incompatible-restore state, and replication stops on the DB instance.A read replica upgrade might fail for one of these reasons:The read replica is unable to catch up with the primary DB instance even after the wait time.The read replica is in a terminal or incompatible lifecycle state, such as storage-full or incompatible-restore.When the primary DB instance upgrade starts, a separate minor version upgrade is running on the read replica.The read replica uses incompatible parameters.The read replica is unable to communicate with the primary DB instance to synchronize the data folder.To resolve this issue, delete the read replica. Then, recreate a new read replica based on the upgraded primary instance after the primary instance is upgraded.Incorrect primary user name: If the primary user name starts with "pg_", then the upgrade fails, and you see the following error message:PreUpgrade checks failed: The instance could not be upgraded because one or more role names start with 'pg_'. Please rename  all roles with names that start with 'pg_' and try againTo resolve this issue, create another user with the rds_superuser role. You can contact AWS Support to update this user as the new primary user.Incompatible parameter error: This error occurs if a memory-related parameter, such as shared_buffer or work_memory, is set to a higher value. This can cause the upgrade script to fail. To fix the issue, reduce the values of these parameters, and then try running the upgrade again.Extensions not updated before the upgrade: A major version upgrade doesn’t upgrade any PostgreSQL extensions. If you didn't update the extensions before performing a major version upgrade, then you see this error in the pg_upgrade.log file:The Logs indicates that the RDS instance ''xxxx'' has older version of PostGIS extension or its dependent extensions (address_standardizer,address_standardizer_data_us, postgis_tiger_geocoder, postgis_topology, postgis_raster) installed as against the current version required for the upgrade.This error message indicates an issue with the PostGIS extension.Run this query to check the default and installed versions for PostGIS and its dependent extensions:SELECT name, default_version, installed_versionFROM pg_available_extensions WHERE installed_version IS NOT NULL ANDname LIKE 'postgis%' OR name LIKE 'address%';If the value for installed_version is less than that of the default_version, then you must update PostGIS to the default version. To do this, run this query:ALTER EXTENSION extension_name UPDATE TO 'default_version_number';For more information, see Upgrading PostgreSQL extensions for RDS for PostgreSQL or Upgrading PostgreSQL extensions for Aurora PostgreSQL.Issue in views due to change in system catalog of the target version: Columns in the certain views vary across different PostgreSQL versions.For example, you might see an error message like this:PreUpgrade checks failed: The instance could not be upgraded because one or more databases have views or materialized views which depend on 'pg_stat_activity'. Please drop them and try again.This error occurs when you upgrade the database from version 9.5 to 9.6. This error is caused due to the pg_stat_activity view because the column waiting is replaced with wait_event_type and wait_event columns in version 9.6.pg_restore: from TOC entry xxx; xxx xxxx VIEW sys_user_constraints art pg_restore: error: could not execute query: ERROR: column c.consrc does not exist LINE 18: "c"."consrc" AS "search_condition", ^ HINT: Perhaps you meant to reference the column "c.conkey" or the column "c.conbin".This error occurs because the structure of the catalog pg_constraint has changed in PostgreSQL version 12.You can resolve these issues by dropping the views based on system catalogs of the target version.Note: Use caution when dropping these views. Be sure to consult with your DBA.Other considerationsThe pg_upgrade utility produces two logs: pg_upgrade_internal.log and pg_upgrade_server.log. Amazon RDS appends a timestamp to the file name for these logs. View these logs to get more information about the issues and errors encountered during the upgrade. For more information, see Monitoring Amazon RDS log files for RDS for PostgreSQL or Monitoring Amazon Aurora log files for Aurora for PostgreSQL.When the upgrade is complete, upgrade the pg_statistics table by running ANALYZE on all user databases. A major upgrade doesn’t move the content of the pg_statistics table to the new version. Skipping this step might result in slow running queries.If you upgraded to PostgreSQL version 10, then run REINDEX on any hash indexes you have. Hash indexes were changed in version 10 and must be rebuilt. To locate invalid hash indexes, run this SQL for each database that contains hash indexes:SELECT idx.indrelid::regclass AS table_name,    idx.indexrelid::regclass AS index_name FROM pg_catalog.pg_index idx   JOIN pg_catalog.pg_class cls ON cls.oid = idx.indexrelid    JOIN pg_catalog.pg_am am ON am.oid = cls.relam WHERE am.amname = 'hash' AND NOT idx.indisvalid;Related informationOverview of upgrading PostgreSQLOverview of the Aurora PostgreSQL upgrade processesFollow"


Question: How can I create an Aurora PostgreSQL-Compatible global database?

I want to create an Amazon Aurora PostgreSQL-Compatible Edition global database.

Answer: "I want to create an Amazon Aurora PostgreSQL-Compatible Edition global database.Short descriptionAurora global databases are used for globally distributed applications. These databases can span multiple AWS Regions, which allows you to have a read-only secondary DB cluster located in another Region to support local reads. Aurora global databases also provide a fast recovery mechanism in the case of an AWS Region failure.Use these methods to create an Aurora PostgreSQL-Compatible global database:Create an Aurora PostgreSQL-Compatible global database using the AWS ConsoleCreate an Aurora PostgreSQL-Compatible global database using an existing Aurora ClusterMigrate an Amazon Relational Database Service (Amazon RDS) for PostgreSQL snapshot to an Aurora PostgreSQL-Compatible global databaseMigrate an Amazon RDS for PostgreSQL database to an Aurora PostgreSQL-Compatible global databaseResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Before creating an Aurora PostgreSQL-Compatible global database:Review the supported engines and Region availability.Review the limitations of Aurora global databases.Review the configuration requirements for Aurora global databases.Create an Aurora PostgreSQL-Compatible global database using the AWS ConsoleOpen the Amazon RDS console in your primary Region.Choose Create database.For Database creation method, choose Standard create.For Engine type, choose Amazon Aurora.For Edition, choose Amazon Aurora PostgreSQL-Compatible Edition.Under filters, turn on Show versions that support the global database feature.In the Available versions dropdown list, choose the version of PostgreSQL that you want to use.Depending on your use case, for Templates, choose Production or Dev/Test.In the Settings section, for DB cluster identifier, enter the name of your primary DB cluster. For Master user name, enter the default master user for PostgreSQL or enter your own user name.In the Instance configuration section, choose a DB instance class.In the Availability durability section, it's a best practice that you choose to create an Aurora replica in a different AZ. If you don't choose that option, you can create the read replica at a later stage.In the Connectivity section, choose the options that you want to use. Also select the VPC, DB Subnet group, and VPC Security group that you want to use.Under Additional configuration you can specify the database port. The default port is 5432.Under Database authentication, by default password authentication is used. You can optionally turn on IAM database authentication, Kerberos authentication, or both for your database.In the Database options section, enter the Initial Database name. Choose the DB cluster parameter group, DB parameter group, and Failover priority. Review the Backup, Encryption, Maintenance, and Deletion Protection options.Choose Create database. When the cluster is ready to use, the status of the cluster, writer, and replica nodes change to Available.An Aurora PostgreSQL global database needs at least one secondary Aurora DB cluster in a different AWS Region than the primary Aurora DB cluster. You can attach up to five secondary DB clusters to your Aurora PostgreSQL global database. Choose the Aurora PostgreSQL database that you created. From the Actions dropdown list, choose Add AWS Region.On the Add a Region page, enter the name for your global database, and choose the secondary Region. Review and complete the rest of the information for the Aurora cluster in the new Region.Note: The names you choose for each of your Aurora DB clusters must be unique, across all AWS Regions. You can't use the same name for different Aurora DB clusters even though they're in different Regions.Click on the Add Region button.Create an Aurora PostgreSQL-Compatible global database using an existing Aurora ClusterOpen the Amazon RDS console in your primary Region.In the Resources section, choose DB Instances.Choose the existing Aurora PostgreSQL cluster that you want to use.From the Actions dropdown list, choose Add AWS Region.On the Add a Region page, enter the name for your global database, and then choose the secondary Region.Choose Add Region.Migrate an RDS for PostgreSQL snapshot to an Aurora PostgreSQL-Compatible global databaseOpen the Amazon RDS console in your primary Region.From the navigation pane, choose Snapshots.Choose the RDS for PostgreSQL snapshot that you want to migrate.Choose Actions, and from the dropdown list choose Migrate Snapshot.On the Migrate database page, for Migrate to DB engine choose aurora-postgresql.Choose the Migrate button. This creates an Aurora PostgreSQL-Compatible cluster from the RDS for PostgreSQL snapshot.After the Aurora PostgreSQL-Compatible cluster is created, from the navigation pane, choose Databases.Choose the migrated Aurora PostgreSQL-Compatible cluster.From the Actions dropdown list, choose Add AWS Region.On the Add a Region page, enter the name for your global database, and then choose the secondary Region.Choose Add Region.Migrate an RDS for PostgreSQL database to an Aurora PostgreSQL-Compatible global databaseOpen the Amazon RDS console in your primary Region.From the navigation pane, choose Databases.Choose the RDS for PostgreSQL database that you want to convert.From the Actions dropdown list, choose Create Aurora read replica.On the Create Aurora read replica page review and complete all fields, and then choose Create read replica.After creating the Aurora read replica, you can now promote the Aurora read replica.Note: Before promoting the replica, make sure that the RDS for PostgreSQL database doesn't have any activity. Confirm also that the replica lag is zero.Choose the Aurora read replica that you created.From the Actions dropdown list, choose promote.After the promotion process, the Aurora replica cluster becomes a Regional Aurora DB cluster with a writer instance. Choose the newly promoted Regional Aurora DB cluster.From the Actions dropdown list, choose Add AWS Region.On the Add a Region page, enter the name for your global database, and then choose the secondary Region.Choose Add Region.Create a headless Aurora DB Cluster in a secondary RegionAurora global databases require at least one secondary Aurora DB Cluster in a different AWS Region. To reduce compute costs, you can create a headless secondary Aurora DB Cluster. By doing this, you are charged only for the storage. While this setup method can incur a higher recovery time objective (RTO), it can reduce the cost of your disaster recover (DR) setup.For PostgreSQL, use the AWS CLI to create the headless Aurora DB Cluster. See this example of a Linux/MacOS AWS CLI command that is used to create a headless Aurora DB Cluster.Note: Replace the <> place holders in this command with actual values before running the command.For a list of options that you can use with this command, see the create-db-cluster reference documentation.aws rds --region <target_region> \  create-db-cluster \     --db-cluster-identifier <value> \     --global-cluster-identifier <value>  \     --engine aurora-postgresql \     --kms-key-id <value> \     --engine-version <value> \     --db-cluster-parameter-group-name <value>\     --db-subnet-group-name <value> \     --vpc-security-group-ids <value> \     --backup-retention-period <value>Related informationAmazon Aurora Global databasesFollow"


Question: How can I troubleshoot connectivity failures and errors for an AWS DMS task that uses Amazon Redshift as the target endpoint?

How can I troubleshoot connectivity failures and errors for an AWS Database Migration Service (AWS DMS) task that uses Amazon Redshift as the target endpoint?

Answer: "How can I troubleshoot connectivity failures and errors for an AWS Database Migration Service (AWS DMS) task that uses Amazon Redshift as the target endpoint?Short descriptionWhen you test the connectivity to an Amazon Redshift endpoint, the test can fail if you haven't met the prerequisites for using an Amazon Redshift database as a target for AWS Database Migration Service. This can happen if you haven't created and configured the required AWS Identity and Access Management (IAM) role or the Amazon Simple Storage Service (Amazon S3) bucket name for an endpoint ARN is in use. The required IAM role is created automatically when you use the AWS DMS console, but it isn't created if you use the AWS DMS API or the AWS Command Line Interface (AWS CLI).A connectivity test can also fail if there are problems with the network configuration of the AWS DMS task. To troubleshoot endpoint connectivity errors, see How can I troubleshoot AWS DMS endpoint connectivity failures?If the required IAM role isn't created and configured correctly, you might receive an error similar to the following:Role 'dms-access-for-endpoint' is not configured properlyResolutionNote: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.Resolve Role 'dms-access-for-endpoint' is not configured properly errorsTo resolve this error, confirm that the dms-access-for-endpoint  IAM role is created and configured correctly. For information about the configuration of this role, see Creating the IAM Roles to use with the AWS CLI and AWS DMS API.If the Amazon managed policy isn't configured correctly, you might receive an error similar to the following:Unable to create S3 bucket for Redshift. Bucket Name for endpoint ARN is in use.This error occurs when:The Amazon managed policy (AmazonDMSRedshiftS3Role or a similar custom policy) isn't attached to the dms-access-for-endpoint IAM role.The dms-access-for-endpoint IAM role policy has an explicit deny for Amazon S3.The preconfigured Amazon S3 bucket policy that AWS DMS created automatically and associated with the Amazon Redshift endpoint has been modified with explicit restriction.To resolve this error, attach the default managed policy (AmazonDMSRedshiftS3Role) or similar custom policy to the dms-access-for-endpoint IAM role. Then, confirm that the default Amazon S3 bucket policy associated by AWS DMS hasn't been modified. For more information, see Amazon S3 bucket settings.Migrate data to an Amazon Redshift endpointWhen you migrate data to an Amazon Redshift target endpoint, DMS uses a default Amazon S3 bucket as intermediate task storage. It then copies the migrated data to Amazon Redshift. When you run the test connection for the target Amazon Redshift endpoint, this automatically creates an S3 bucket with the following naming convention:dms-'Redshift endpoint ARN'You can choose a custom S3 bucket for this intermediate storage. For more information, see Using an Amazon Redshift database as a target for AWS Database Migration Service.Resolve Amazon S3; Status Code: 400; Error Code: TooManyBuckets errorsIf your account has reached the limit for Amazon S3, you might receive an error similar to the following when you test your endpoint:Service: Amazon S3; Status Code: 400; Error Code: TooManyBuckets; Request ID: xxxxxxxxxxx; S3 Extended Request ID: xxxxxxxxxxxxxx; Proxy: nullTo resolve this error, delete unused buckets from your account, and test the endpoint again.Resolve <NoSuchBucket> The specified bucket does not exist errorsIf you delete an Amazon S3 bucket that is created by AWS DMS during task migration, you might receive an error similar to the following in the task logs:<NoSuchBucket> The specified bucket does not exist.To resolve this issue, test the connection for your Amazon Redshift endpoint, and then restart or resume the task. If you have configured your DMS endpoint to use as a custom bucket, then make sure that it is available in Amazon S3 before starting the task again.Related informationUsing an Amazon Redshift database as a target for AWS Database Migration ServiceFollow"


Question: How can I use custom policy rules with my Firewall Manager content audit security group policy?

I want to use custom policy rules with my AWS Firewall Manager content audit security group policy.

Answer: "I want to use custom policy rules with my AWS Firewall Manager content audit security group policy.Short descriptionYou can use Firewall Manager content audit security group policies to check and manage rules that are in use in your organization's security groups. Content audit security group policies apply to all security groups in use in your organization in AWS Organizations. You can create custom policy rules for your content audit security group policy for your use case.For more information, see Content audit security group policies.ResolutionComplete the Firewall Manager prerequisites, and then create the audit security group and policyBefore you begin, complete the Firewall Manager prerequisites.Create an audit security group with your Firewall Manager administrator account. Follow the best practices for security group policies and the security group policy limitations.Follow the steps for creating Firewall Manager content audit security group policy.Important: It's a best practice to create an audit security group policy with automatic remediation deactivated. Review the effects of policy creation before activating automatic remediation. After you review the expected effects, you can edit the policy and then activate automatic remediation. When automatic remediation is activated, Firewall Manager updates or removes rules that are noncompliant with in-scope security groups."Allow only the rules defined in the audit security groups" policy ruleThis rule states that all in-scope security groups must only have rules that are within the allowed range of the policy's audit security group rules. In this case, the policy's security group rules provide the example of what's acceptable to do.Example use caseTo allow SSH from allow-listed CIDRs in the range 10.0.0.0/16, use an inbound rule allowing SSH (TCP port 22) from 10.0.0.0/16 only. Security group rules allowing SSH from other CIDRs in another range (for example, 10.0.0.0/8) aren't allowed.For more information, see Content audit security group policies."Deny the use of any rules defined in the audit security groups" policy ruleThis rule states that all in-scope security groups must only have rules that aren't within the allowed range of the policy's audit security group rules. In this case, the policy's security group provides the example of what's not acceptable to do.Example use caseTo deny inbound TCP access on the port range 1024 to 65535, use an inbound rule allowing TCP traffic in the range 0.0.0.0/0. Security group rules allowing TCP traffic in the port range 1024 to 65535 from any CIDR aren't allowed.Related informationHow do I set up AWS Firewall Manager for my AWS account?Follow"


Question: How do I reduce my AWS bill?

I want to reduce my AWS bill. What are the different cost saving options available?

Answer: "I want to reduce my AWS bill. What are the different cost saving options available?Short descriptionYou can use a variety of tools and approaches to analyze and optimize the costs of your AWS resources. You can choose the cost optimization techniques that are appropriate for your business and workload.ResolutionAnalyze your costsUse Cost Explorer to view, analyze, and forecast your spending and usage free of charge. Cost Explorer provides historical data, current month cost data, and forecasts for the next three months.Create an AWS Cost and Usage Report to get detailed data about your AWS bill in one of your Amazon Simple Storage Service (Amazon S3) buckets. For information on how to use this report, see How do I use the AWS Cost and Usage Report?Use AWS Budgets to plan your service usage, costs, and instance reservations. With AWS Budgets, you can set custom budgets that proactively alert you when your costs exceed your budgeted amount. For more information, see How can I use AWS Budgets to track my spending and usage?Optimize your costsNote: These tools can help you manage your AWS usage and costs, but they cannot automatically pause or stop AWS services.AWS Trusted Advisor can help you identify resources that you aren't using to their full capacity. Then, you can decide to delete these underutilized resources to optimize your costs. To receive email notifications of Trusted Advisor checks, see How do I set up email notifications for AWS Trusted Advisor in my account? For information on how to delete your underutilized resources, see How do I terminate active resources that I no longer need on my AWS account?If you have predictable workloads on Amazon Elastic Compute Cloud (Amazon EC2), AWS Fargate, or AWS Lambda and are paying the default On-Demand pricing, then you can save money by choosing an appropriate Savings Plan. For more information, see How do I save money on my AWS usage using a Savings Plan?Use the AWS Instance Scheduler to reduce the costs of EC2 and Amazon Relational Database Service (Amazon RDS) instances used in non-production environments. For more information, see How do I use Instance Scheduler with CloudFormation to schedule EC2 instances?If you're using the S3 Standard storage class and have infrequently accessed objects, then enable S3 Intelligent-Tiering for automatic cost optimization. You can choose S3 Intelligent-Tiering when you upload your objects to Amazon S3. Then, add the appropriate Lifecycle configuration to move your objects to S3 Intelligent-Tiering.Use Amazon EC2 Spot Instances for your stateless, fault-tolerant, or flexible applications, such as containerized workloads, to reduce your operating costs. For more information, see Spot Instances.When EC2 instances are terminated, the attached Amazon Elastic Block Store (Amazon EBS) volumes aren't automatically deleted. Find the unattached EBS volumes that you no longer need. Then, you can delete them after optionally creating an EBS snapshot. For more information, see Why am I being charged for Amazon EBS when all my instances are stopped?Important: The data in the EBS volume is lost on deletion, and the volume can't be attached to any instance.Related informationSix ways to reduce your AWS billOptimize and save your IT costsFollow"


Question: How do I add new user accounts with SSH access to my Amazon EC2 Linux instance?

I want to add new user accounts that can connect to my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance using SSH.

Answer: "I want to add new user accounts that can connect to my Amazon Elastic Compute Cloud (Amazon EC2) Linux instance using SSH.Short descriptionEvery Amazon EC2 Linux instance launches with a default system user account with administrative access to the instance. If multiple users require access to the instance, then it's a security best practice to use separate accounts for each user.You can expedite these steps by using cloud-init and user data. For more information, see How do I add new user accounts with SSH access to my EC2 instance using cloud-init and user data?ResolutionCreate a key pair for the new user accountCreate a key pair, or use an existing one, for the new user.If you create your own key pair using the command line, then follow the recommendations at create-key-pair or New-EC2KeyPair Cmdlet for key type and bit length.If you create your own key pair using a third-party tool, then be sure that your key matches the guidelines. For more information, see Create a key pair using a third-party tool and import the public key to Amazon EC2.Add a new user to the EC2 Linux instance1.    Connect to your Linux instance using SSH.2.    Use the adduser command to add a new user account to an EC2 instance (replace new_user with the new account name). The following example creates an associated group, home directory, and an entry in the /etc/passwd file of the instance.$ sudo adduser new_userThe home directory might not be created by default in some configurations. Verify that the home directory was created before continuing.Note: If you add the new_user to an Ubuntu instance, then include the --disabled-password option to avoid adding a password to the new account:$ sudo adduser new_user --disabled-password3.    Change the security context to the new_user account so that folders and files you create have the correct permissions:$ sudo su - new_userNote: When you run the sudo su - new_user command, the name at the top of the command shell prompt changes to reflect the new user account context of your shell session.4.    Create a .ssh directory in the new_user home directory:$ mkdir .ssh5.    Use the chmod command to change the .ssh directory's permissions to 700. Changing the permissions restricts access so that only the new_user can read, write, or open the .ssh directory.$ chmod 700 .ssh6.    Use the touch command to create the authorized_keys file in the .ssh directory:$ touch .ssh/authorized_keys7.    Use the chmod command to change the .ssh/authorized_keys file permissions to 600. Changing the file permissions restricts read or write access to the new_user.$ chmod 600 .ssh/authorized_keysRetrieve the public key for your key pairRetrieve the public key for your key pair using the method that applies to your configuration:Create a key pair using a third-party tool and import the public key to Amazon EC2Retrieve the public key from the private keyRetrieve the public key for your key pair through instance metadataVerify your key pair's fingerprintAfter you import your own public key or retrieve the public key for your key pair, follow the steps at Verify your key pair's fingerprint.Update and verify the new user account credentialsAfter you retrieve the public key, confirm that you have permission to add the public key to the .ssh/authorized_keys file for this account:1.    Run the Linux cat command in append mode:$ cat >> .ssh/authorized_keys2.    Paste the public key into the .ssh/authorized_keys file and then press Enter.Note: For most Linux command line interfaces, the Ctrl+Shift+V key combination pastes the contents of the clipboard into the command line window. For the PuTTY command line interface, right-click to paste the contents of the clipboard into the PuTTY command line window.3.    Press and hold Ctrl+d to exit cat and return to the command line session prompt.Verify that the new user can use SSH to connect to the EC2 instance1.    Run the following command from a command line prompt on your local computer:$ ssh -i /path/new_key_pair.pem new_user@public_dns_name_of_EC2_Linux_instanceTo connect to your EC2 Linux instance using SSH from Windows, follow the steps at Connect to your Linux instance from Windows using PuTTY.Note: If you receive errors when trying to connect, refer to Troubleshoot connecting to your instance.2.    Run the id command from the instance's command line to view the user and group information created for the new_user account:$ idThe id command returns information similar to the following:uid=1004(new_user) gid=1004(new_user) groups=1004(new_user)3.    Distribute the private key file to your new user.Related informationManage users on your Linux instanceFollow"


Question: How do I redirect a domain in CloudFront?

I want to redirect my domain to another domain in Amazon CloudFront. How can I do this?

Answer: "I want to redirect my domain to another domain in Amazon CloudFront. How can I do this?Short descriptionTo redirect a domain in CloudFront, use one of the following:An Amazon Simple Storage Service (Amazon S3) website endpoint that returns a 301 status codeAn edge function that redirects requests to the new domainAn Amazon S3 website endpoint redirection moves the existing URL path to the redirected domain. Edge functions allow you to redirect viewers to other pages based on information in the request, or rewrite all requests from one path to another.ResolutionUse an Amazon S3 endpoint that returns a 301 status codeFollow the instructions in theRedirect using Route 53 and Amazon S3 section of How can I redirect one domain to another in Route 53?Use an edge function that redirects requests to the new domainFor more information on the difference between Lambda@Edge and CloudFront functions, see Choosing between CloudFront Functions and Lambda@Edge.For Lambda@Edge:See Customizing at the edge with Lambda@Edge. For example code, see Example: Generating an HTTP redirect (generated response).For CloudFront Functions:See Customizing at the edge with CloudFront Functions. For example code, see Redirect the viewer to a new URL.Related informationConfiguring a webpage redirectCreating a Lambda@Edge function in the Lambda consoleTutorial: Creating a simple function with CloudFront FunctionsFollow"


Question: How do I influence my Direct Connect connection network traffic path to on-premises over multiple circuits?

I want to influence my AWS Direct Connect connection network traffic path to on-premises over multiple circuits.

Answer: "I want to influence my AWS Direct Connect connection network traffic path to on-premises over multiple circuits.Short descriptionYou can use multiple Direct Connect circuits in different AWS Regions to increase bandwidth and high availability. Depending on the distribution of the circuits across Regions, you can influence Direct Connect traffic by advertising Border Gateway Protocol (BGP) community tags with on-premises prefixes.Common setups for multiple Direct Connect connections include:Private virtual interfaces created on the Direct Connect connections attached to the same virtual private gateways (VGW) in the same Region.Private virtual Interfaces created on the Direct Connect connections attached to the same Direct Connect Gateway (DXGW) and associated with multiple VGWs in any Regions.Transit virtual Interfaces created on the Direct Connect connections attached to the same DXGW and associated with multiple transit gateways in any Regions.ResolutionFollow these instructions to influence your Direct Connect connection network traffic path based on your use case.Default behavior for private and transit virtual Interfaces and how to influence itTo influence the default behavior, it's a best practice to useAutonomous System (AS) path prepending and the following local preference BGP community tags:7224:7100—Low preference7224:7200—Medium preference7224:7300—High preferenceLocal preference BGP community tags are evaluated in order from lowest to highest preference (where highest preference is preferred). For each prefix that you advertise over a BGP session, you can apply a community tag to indicate the priority of the associated path for returning traffic.For more information, see How can I use BGP communities to influence the preferred routing path on Direct Connect links from AWS to my network?Scenario 1You have one Direct Connect connection (DX1) in the us-east-1 Region, and a second connection (DX2) in the eu-west-1 Region. A private virtual interface (VIF) is created on DX1 and DX2 with the same prefix's advertised from on-premises devices. DX1 and DX2 VIF's are attached with a DXGW associated with three VGWs in the us-east-1, eu-west-1, and ap-southeast-1 Regions.If you advertise BGP tag 7224:7300 over the VIF on DX1, traffic from the us-east-1 Region Amazon Virtual Private Cloud (Amazon VPC) is unchanged. Traffic from the eu-west-1 and ap-southeast-1 Regions prefer DX1 because the same region preference is overwritten by the BGP community.If you advertise BGP tag 7224:7300 over VIFs on DX1 and DX2, traffic from all Amazon VPCs in different regions are load balanced across DX1 and DX2. This is because the same region preference is overwritten by the BGP community.If you advertise BGP tag 7224:7300 over both VIFs with AS prepending on DX1, traffic from all Amazon VPCs in all regions prefer DX2. This is because the region preferences are overwritten as equal. Then, AWS performs validation on the AS path length finding that the path prefix for DX2 is shorter than DX1.If you only advertise one AS prepending over VIF on DX1 without BGP community tags, traffic from Amazon VPCs in the us-east-1 and eu-west-1 Regions remains unchanged. Traffic from the ap-southeast-1 Region prefers DX2. This is because the region preference has a higher priority than the AS path length.Scenario 2You have Direct Connect connections DX1 and DX2 in the us-east-1 Region with private virtual interfaces using the same prefix advertised from on-premises devices. The private virtual interfaces are attached with the DXGW associated with three VGWs in the us-east-1 and eu-west-1 Regions.If you advertise BGP tag 7224:7300 over the VIF on DX1, traffic from the us-east-1 and eu-west-1 Regions prefer DX1 because the same region preference is overwritten by the BGP community.If you advertise BGP tag 7224:7300 over both VIFs on DX1 and DX2, traffic from VPCs in different regions remain unchanged. This is because the region preference is overwritten by the BGP community tag for load balancing.If you advertise BGP tag 7224:7300 over both VIFs with AS prepending on DX1, traffic from Amazon VPCs in all regions prefers DX2. This is because the region preference is overwritten as equal. Then, AWS performs validation on the AS path length finding that the path prefix for DX2 is shorter than DX1.If you advertise one AS prepending over the VIF on DX1 without BGP community tags, traffic from Amazon VPCs in the us-east-1 and eu-west-1 Regions prefers DX2. This is because the us-east-1 and eu-west-1 Regions have the same region preference and a shorter AS path for DX2.Scenario 1 and Scenario 2If you advertise on-premises prefixes without influencing factors, the default traffic egress behavior is the following:Traffic sourced from the Amazon VPC in the us-east-1 Region is preferred for DX1 because it’s in the same region. Traffic sourced from the Amazon VPC in the eu-west-1 Region is preferred for DX2 because it's in the same region. Because the ap-southeast-1 is a remote region, traffic sourced from the Amazon VPC is load balanced across DX1 and DX2.Traffic sourced from the Amazon VPC in the us-east-1 Region is load balanced across DX1 and DX2 because they are in the same region. Traffic sourced from the Amazon VPC in the eu-west-1 Region is load balanced across DX1 and DX2.Default behavior for public virtual interfaces and how to influence itIn the following scenario, you have two Direct Connect connections in the same Region that use the same public virtual interface and prefix advertised from on- premises devices.Egress traffic from the AWS direction is load balanced across public VIFs without influencing factors. To influence the default behavior, It's a best practice to use Autonomous System (AS) prepending with a public ASN. If you can only use a private ASN, it's a best practice to advertise more specific prefixes from the preferred public VIF, so that the longest prefix match determines the routing path.Common asymmetric routing scenarios for public virtual interfacesYou can use the following BGP prefixes with Direct Connect:7224:9100—Local AWS Region7224:9200—All AWS Regions for a continent7224:9300—Global (all public AWS Regions)For more information, see the scope BGP communities.In the following scenario, you have a Direct Connect connection in the us-east-1 Region with a public virtual interface. You're advertising the same on-premises public prefix with the community tag 7224:9100 over the Direct Connect connection and the local internet service provider.If you're accessing an Amazon Simple Storage Service (Amazon S3) bucket in the us-east-1 Region, the inbound and outbound traffic is over the Direct Connect connection.If you're accessing an Amazon S3 bucket in the eu-west-1 Region, the inbound traffic is over the Direct Connect connection, and the outbound traffic is over the local internet service provider. This is because the prefix isn't propagated to the eu-west-1 Region causing asymmetric routing.If you’re accessing an Amazon S3 bucket in the us-east-1 Region and the inbound traffic from on-premises uses the local internet service provider, asymmetric routing occurs. This is because AWS prefers to send outbound traffic over the Direct Connect connection instead of the local internet service provider when the same prefixes are received.Related informationConfigure redundant connections with AWS Direct ConnectHow do I set up an Active/Active or Active/Passive Direct Connect connection to AWS from a public virtual interface? AWS Direct Connect virtual interface typesFollow"


Question: How do I resolve the "Unable to infer schema" exception in AWS Glue?

My AWS Glue job fails with one of the following exceptions:"AnalysisException: u'Unable to infer schema for Parquet. It must be specified manually.;'""AnalysisException: u'Unable to infer schema for ORC. It must be specified manually.;'"

Answer: "My AWS Glue job fails with one of the following exceptions:"AnalysisException: u'Unable to infer schema for Parquet. It must be specified manually.;'""AnalysisException: u'Unable to infer schema for ORC. It must be specified manually.;'"Short descriptionThis error usually happens when AWS Glue tries to read a Parquet or Orc file that's not stored in an Apache Hive-style partitioned path with the key=val structure. AWS Glue expects the Amazon Simple Storage Service (Amazon S3) source files to be in key-value pairs. For example, if the AWS Glue job is processing files at s3://s3-bucket/parquet-data/, the files must have the following partition structure:s3://s3-bucket/parquet-data/year=2018/month=10/day=10/file1.parquetIf your Parquet or Orc files are stored in a hierarchical structure, the AWS Glue job fails with the "Unable to infer schema" exception. Example:s3://s3-bucket/parquet-data/year/month/day/file1.parquetResolutionUse one of the following methods to resolve the error.Restructure your dataCopy the files into a new S3 bucket and use Hive-style partitioned paths. Run the job again.Replace partition column names with asterisksIf restructuring your data isn't feasible, create the DynamicFrame directly from Amazon S3. In the Amazon S3 path, replace all partition column names with asterisks (*). When you use this solution, AWS Glue does not include the partition columns in the DynamicFrame—it only includes the data.For example, assume that your files are stored in an S3 bucket with the following partition structure:s3://s3-bucket/parquet-data/year/month/day/files.parquetTo process all files in the s3://s3-bucket/parquet-data/ path, create the DynamicFrame:dynamic_frame0 = glueContext.create_dynamic_frame_from_options('s3',connection_options={'paths':['s3://s3-bucket/parquet-data/*/*/*'],},format="parquet",transformation_ctx = "dynamic_frame0")Use a map transformation to add partition columnsTo include the partition columns in the DynamicFrame, create a DataFrame first, and then add a column for the Amazon S3 file path. Then, create the DynamicFrame and apply a map transformation to add the partition columns, as shown in the following example. Before using the sample code, replace the Amazon S3 paths and enter your partition column names using the correct index values.import sysfrom awsglue.transforms import *from awsglue.utils import getResolvedOptionsfrom pyspark.context import SparkContextfrom awsglue.context import GlueContextfrom awsglue.job import Jobfrom awsglue.dynamicframe import DynamicFramefrom pyspark.sql.functions import *args = getResolvedOptions(sys.argv, ['JOB_NAME'])sc = SparkContext()glueContext = GlueContext(sc)spark = glueContext.spark_sessionjob = Job(glueContext)job.init(args['JOB_NAME'], args)df= spark.read.parquet("s3://s3-bucket/parquet-data/*/*/*")modified_df = df.withColumn('partitions_column',input_file_name())dyf_0 = DynamicFrame.fromDF(modified_df, glueContext, "dyf_0")def modify_col(x):     if x['partitions_column']:        new_columns = x['partitions_column'].split('/')        x['year'],x['month'],x['day'] = new_columns[4],new_columns[5],new_columns[6]        del x['partitions_column']     return xmodified_dyf = Map.apply(dyf_0,f=modify_col)datasink2 = glueContext.write_dynamic_frame.from_options(frame =modified_dyf , connection_type = "s3", connection_options = {"path": "s3://my-output-bucket/output/","partitionKeys": ["year","month","day"]}, format = "parquet", transformation_ctx = "datasink2")For more information about applying map transformations, see Map Class.Related informationManaging partitions for ETL output in AWS GlueFollow"


Question: How do I resolve the manual snapshot error in my OpenSearch Service cluster?

"I tried to restore a manual snapshot for my Amazon OpenSearch Service cluster, but I received an error when trying to register or access a repository."

Answer: "I tried to restore a manual snapshot for my Amazon OpenSearch Service cluster, but I received an error when trying to register or access a repository.Short descriptionTo migrate data from a manual snapshot in OpenSearch Service, complete the following steps:1.    Choose an Amazon Simple Storage Service (Amazon S3) bucket where you want to store your snapshot.2.    Register the Amazon S3 bucket with your OpenSearch Service source cluster.3.    Take a snapshot of the OpenSearch Service source cluster, and then store it in your Amazon S3 bucket.4.    Register your destination cluster with the same Amazon S3 bucket to make sure that you can view the manual snapshot.5.    Restore the manual snapshot on the destination cluster in OpenSearch Service.Otherwise, you might encounter one of the following errors:403 Unauthorized errorillegal_state_exceptionrepository_missing_exceptionconcurrent_snapshot_execution_exceptionsnapshot_restore_exceptiona_w_s_security_token_service_exception"PARTIAL" snapshot statusAmazon Simple Storage Service Glacier (Amazon S3 Glacier) storage class issueResolution403 Unauthorized errorIf you activated fine-grained access control (FGAC) on your OpenSearch Service domain, then you might receive the following error when you take a snapshot:{    "error": {        "root_cause": [{            "type": "security_exception",            "reason": "no permissions for [cluster:admin/repository/put] and User [name=arn:aws:iam::012345678912:user/username, backend_roles=[], requestedTenant=null]"    }],        "type": "security_exception",        "reason": "no permissions for [cluster:admin/repository/put] and User [name=arn:aws:iam::012345678912:user/username, backend_roles=[], requestedTenant=null]"    },    "status": 403}To resolve the 403 unauthorized error, make sure to specify a username:password parameter when you take a manual snapshot:curl -XPUT -u 'opensearch-domain-endpoint/_snapshot/snapshot-repository-name/snapshot-name'Note: You must be a superuser to activate fine-grained access control for your OpenSearch Service domain. You can either use your superuser name and password, or set an AWS Identity Access Management (IAM) role as the superuser. When you access your cluster snapshot, specify your superuser credentials or IAM role. If you specify an IAM role, then the IAM role must use sigv4 to sign the HTTP requests. For more information about using fine-grained access control and IAM roles, see Creating and managing OpenSearch Service domains.You must also register a snapshot repository with your snapshot, and map the manage_snapshots role to an IAM role. The manage_snapshots role must have valid permissions (IAM:PassRole) to assume the IAM role (TheSnapshotRole). For more information, see Prerequisites.To map the manage_snapshots role to an IAM role, complete the following steps:1.    Open the OpenSearch Service console as a primary user.2.    Choose Security.3.    Choose Roles.4.    Choose manage_snapshots as your role.5.    Choose Mapped users.6.    Choose Manage mapping.7.    Under Users, add your user ARN (for example: "arn:aws:iam::012345678912:user/username").8.    Register your manual snapshot repository.illegal_state_exceptionThe following error occurs when you use the Amazon S3 bucket for multiple domains for taking manual snapshots:{  "error": {    "root_cause": [      {        "type": "illegal_state_exception",        "reason": "Can't get text on a FIELD_NAME at 1:1838"      }    ],    "type": "illegal_state_exception",    "reason": "Can't get text on a FIELD_NAME at 1:1838"  },  "status": 500}To resolve this issue, create a new Amazon S3 bucket and take manual snapshot, or clear all of the data from the existing bucket. Repository_missing_exceptionBefore you take a manual index snapshot, you must register a manual snapshot repository with OpenSearch Service. You must also set up your IAM role (TheSnapshotRole) to work with Amazon S3.If you didn't register your snapshot repository before taking a manual snapshot, or you use an incorrect repository name, then you receive the following error:{    "error": {        "root_cause": [{            "type": "repository_missing_exception",            "reason": "[snapshot-repository-name] missing"        }],        "type": "repository_missing_exception",        "reason": "[snapshot-repository-name] missing"    },    "status": 404}To resolve this error, make sure that you meet the manual snapshot prerequisites. Also, make sure that there aren't any typos in the repository name.Concurrent_snapshot_execution_exceptionIf a snapshot is in progress, then you receive the following error when you try to take another snapshot:{    "error": {        "root_cause": [{            "type": "concurrent_snapshot_execution_exception",            "reason": "[snapshot-repository-name:snapshot-name] a snapshot is already running"        }],        "type": "concurrent_snapshot_execution_exception",        "reason": "[snapshot-repository-name:snapshot-name] a snapshot is already running"    }}To check if there's another snapshot in progress, run the following command:curl -XGET 'opensearch-domain-endpoint/_snapshot/_status'If a snapshot is already in progress, then wait for the current snapshot to complete. Or, if you think that your snapshot is stuck, then check your history of hourly snapshots. For more information, see Why can't I delete an index or upgrade my OpenSearch Service cluster?Snapshot_restore_exceptionIf you try to migrate data from an on-premises cluster to an OpenSearch Service domain, then you might encounter the following exception error:{    "error": {        "root_cause": [{            "type": "snapshot_restore_exception",            "reason": "[manual-snapshot-repo:my-manual-snapshot1/HPOcIJryTj6a6GJvyP79bw] the snapshot was created with Elasticsearch version [6.8.0] which is higher than the version of this node [6.7.0]"        }],        "type": "snapshot_restore_exception",        "reason": "[manual-snapshot-repo:my-manual-snapshot1/HPOcIJryTj6a6GJvyP79bw] the snapshot was created with Elasticsearch version [6.8.0] which is higher than the version of this node [6.7.0]"    },    "status": 500}This error message occurs when a snapshot that you take on an existing cluster runs on a different version of Elasticsearch than OpenSearch Service. If your cluster is running on an earlier version of Elasticsearch than OpenSearch Service, then upgrade your Elasticsearch version. Or, you can use the remote reindex API to migrate your indices.If the domain's FGAC is activated and you try to restore all the indices from the snapshot, then you might receive a 403 error similar to the following one:{  "error" : {    "root_cause" : [      {        "type" : "security_exception",        "reason" : "no permissions for [] and User [name=RahulSarkar, backend_roles=[], requestedTenant=]"      }    ],    "type" : "security_exception",    "reason" : "no permissions for [] and User [name=RahulSarkar, backend_roles=[], requestedTenant=]"  },  "status" : 403}To resolve this error, exclude the security indices similar to the following ones:curl -XPOST 'username:password' "https://opensearch-domain-endpoint/_snapshot/snapshot-repository/snapshot-id/_restore" -H 'Content-Type: application/json' -d'{  "indices": "-.opensearch*,-.opendistro*,-.kibana*"}'a_w_s_security_token_service_exceptionIf the IAM role that's associated with your manual snapshot doesn't have a trust relationship established for "es.amazonaws.com", then you receive the following exception error:{    "error": {        "root_cause": [{            "type": "repository_exception",            "reason": "[es_01082021_repo] Could not determine repository generation from root blobs"        }],        "type": "repository_exception",        "reason": "[es_01082021_repo] Could not determine repository generation from root blobs",        "caused_by": {            "type": "i_o_exception",            "reason": "Exception when listing blobs by prefix [index-]",            "caused_by": {                "type": "a_w_s_security_token_service_exception",                "reason": "a_w_s_security_token_service_exception: User: arn:aws:sts::332315457451:assumed-role/cp-sts-grant-role/swift-us-west-2-prod-679203657591 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::679203657591:role/ES_Backup_Role (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 36d09b93-d94f-457e-8fa5-b0a50ba436c3)"            }        }    },    "status": 500}With OpenSearch Service snapshots, an internal role is created (such as arn:aws:sts::332315457451:assumed-role/cp-sts-grant-role/swift-us-west-2-prod-679203657591). This internal role assumes the IAM role that's associated with the manual snapshot, and then performs any required operations.To resolve the security token exception error, make sure to specify the IAM role that's associated with the manual snapshot. If you don't have an IAM role that's associated with the manual snapshot, then create one. For more information, see Prerequisites.Also, check the trust relationship for the IAM role that's associated with the manual snapshot. The trust relationship for the role must specify OpenSearch Service in the Principal statement, similar to this one:{    "Version": "2012-10-17",    "Statement": [{        "Sid": "",        "Effect": "Allow",        "Principal": {            "Service": "es.amazonaws.com"        },        "Action": "sts:AssumeRole"    }]}PARTIAL snapshot statusA snapshot enters a "PARTIAL" state for the following reasons:Unavailable primary shards in your clusterRed cluster statusNode crashA partial snapshot indicates that data from a shard can't be stored. You can still restore data from a partial snapshot, but you must use earlier snapshots to restore any missing indices. To check whether your cluster entered a "PARTIAL" state, check your snapshot history. For more information, see Restoring snapshots.Amazon S3 Glacier storage classes issueIf you're storing a restored snapshot in one of the S3 Glacier storage classes, then don't apply an Amazon S3 Glacier Lifecycle rule to the bucket. Manual snapshots don't support the S3 Glacier storage classes. Therefore, if you apply an Amazon S3 Glacier Lifecycle policy to the S3 bucket, then you must move back any objects that transition over.After you move the objects back to a standard Amazon S3 storage class, you can restore the objects from those snapshots. For more information, see Prerequisites. Related informationHow do I resolve the "cannot restore index [.kibana]" error in Amazon OpenSearch Service?Taking manual snapshotsFollow"


Question: How do I change or remove the default message played when transferring a call to a queue in Amazon Connect?

"When agents in my Amazon Connect contact center transfer a call to a queue, I don't want my customers to hear the default "Thank you for calling" message. How do I change or remove it?"

Answer: "When agents in my Amazon Connect contact center transfer a call to a queue, I don't want my customers to hear the default "Thank you for calling" message. How do I change or remove it?Short descriptionWhen calls in your contact center are transferred to a queue, the default customer queue contact flow runs by default. In this contact flow, the following audio prompt is played:"Thank you for calling. Your call is very important to us and will be answered in the order it was received."To play a different message or no message at all, route calls into another customer queue contact flow that you design.ResolutionCreate a customer queue contact flowImportant: To create a contact flow, you must log in to your Amazon Connect instance as a user with sufficient permissions in their security profile.1.    Log in to your Amazon Connect instance.Note: Replace the alias with your instance's alias.2.    In the left navigation bar, pause on Routing. Then, choose Contact flows.3.    On the Contact flows page, next to the Create contact flow button, choose the down arrow. Then, choose Create customer queue flow.4.    In the contact flow designer, for Enter a name, enter a name for the contact flow. (For example, Custom queue audio.)5.    Choose Save.For more information, see Create a new contact flow.Edit your new contact flowIn your new customer queue flow, do one of the following to either set up a custom audio message or have no audio message:To set up a custom audio messageNote: Using this customer queue flow, a custom audio file or spoken message is played in the queue before an agent answers a call.1.    Choose Interact. Then, drag and drop a Loop prompts block onto the canvas.2.    Choose the block title (Loop prompts). The block's settings menu opens.3.    Choose Add another prompt to the loop to add an audio prompt.4.    Under Prompts, do one of the following:Configure the prompt as an Audio recording. You can choose a default audio prompt (for example, CustomerHold.wav) or an audio prompt that you create.-or-Configure the prompt as Text to speech with a custom spoken message. For more information, see Add text-to-speech to prompts.5.    (Optional) Repeat steps 3 and 4 to add more audio prompts to the loop.6.    Choose Save.7.    (Optional) In the contact flow designer, add and then configure any other contact blocks required for your use case.8.    In the contact flow designer, choose Save.9.    In the Save flow dialog, choose Save.10.    Choose Publish.11.    In the Publish dialog, choose Publish.To set up no audio messageNote: Using this customer queue flow, no audio is played in the queue before an agent answers a call.1.    Choose Terminate / Transfer. Then, drag and drop an End flow / Resume block onto the canvas.2.    On the Entry point block, drag the arrow from Start to the End flow / Resume block.3.    Choose Save.4.    In the Save flow dialog, choose Save.5.    Choose Publish.6.    In the Publish dialog, choose Publish.Edit your main contact flow1.    In the left navigation bar, pause on Routing. Then, choose Contact flows.2.    On the Contact flows page, choose your main contact flow.3.    In the contact flow designer, choose Set. Then, drag and drop a Set customer queue flow block onto the canvas.4.    Choose the block title (Set customer queue flow). The block's settings menu opens.5.    Under Customer queue flow, choose Select a flow.6.    In the Search for contact flow dropdown list that appears, choose the new customer queue flow that you created.7.    Choose Save.8.    In the contact flow designer, connect the Set customer queue flow block after the Set working queue block and before the Transfer to queue block. For example:Set working queue > Set customer queue flow > Transfer to queue9.    Choose Save.10.    In the Save flow dialog, choose Save.11.    Choose Publish.12.    In the Publish dialog, choose Publish.Related informationSet up contact transfersFollow"


Question: How do I associate a Route 53 private hosted zone with a VPC on a different AWS account?

I want to associate my Amazon Route 53 private hosted zone with a virtual private cloud (VPC) that belongs to a different AWS account.

Answer: "I want to associate my Amazon Route 53 private hosted zone with a virtual private cloud (VPC) that belongs to a different AWS account.ResolutionIn the following resolution, use one of these options to run the commands:Option 1: AWS Command Line Interface (AWS CLI). The resolution uses two example Amazon Elastic Compute Cloud (Amazon EC2) instances, one in Account A and the other in Account B. If you don't have two Amazon EC2 instances or don't have access to launch new EC2 instances, then use the AWS CLI on your local machine. Use the correct AWS Identity and Access Management (IAM) credentials from both accounts. For more information, see Using an IAM Identity Center activated named profile.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.Option 2: AWS CloudShell (without installing AWS CLI).Note: You can also use the AWS SDK or Route 53 API for this process.1.    Connect to an EC2 instance in Account A.2.    If using the AWS CLI, then run the following command to update the AWS CLI version. Configure the AWS CLI to use the credentials of an AWS Identity and Access Management (IAM) user that has Route 53 access.pip3 install awscli --upgrade --user3.    In the EC2 instance in Account A, run the following command to list the available hosted zones. Note the hosted zone ID in Account A that you want to associate with Account B.aws route53 list-hosted-zones4.    In the EC2 instance in Account A, run the following command. The command output lists the VPCs from other accounts that you can associate with your private hosted zone. The accounts listed in the command output are those accounts that you submitted one or more CreateVPCAssociationAuthorization requests for.aws route53 list-vpc-association-authorizations --hosted-zone-id <hosted-zone-id>Note: If the VPC in Account B doesn't appear in the VPC association authorizations list, then proceed to step 5. If it does appear in the list, then proceed to step 6.5.    In the EC2 instance in Account A, run the following command. This command authorizes the association between the private hosted zone in Account A and the VPC in Account B. In the following command, use the hosted zone ID that you obtained in the previous step. Use the AWS Region and ID of the VPC in Account B.Note: If one of the following scenarios is true, then use the --region in the command:You're running the command from an EC2 instances in a different Region.The user's credentials are associated with a Region other than us-east-1.aws route53 create-vpc-association-authorization --hosted-zone-id <hosted-zone-id> --vpc VPCRegion=<region>,VPCId=<vpc-id> --region us-east-16.    Connect to an EC2 instance in Account B.7.    In the EC2 instance in Account B, run the following command. This command creates the association between the private hosted zone in Account A and the VPC in Account B. Use the hosted zone ID from step 3. Use the Region and ID of the VPC in Account B.Note: Use an IAM user or role that has AssociateVPCWithHostedZone and DescribeVpcs API permissions to run the following command in Account B.aws route53 associate-vpc-with-hosted-zone --hosted-zone-id <hosted-zone-id> --vpc VPCRegion=<region>,VPCId=<vpc-id> --region us-east-1Note: The sample output might show a status of PENDING. This occurs because the VPC can't use the private hosted zone to perform DNS resolution. It might take a few minutes for the private hosted zone to associate with the VPC and for the changes to propagate./p>8.    It's a best practice to delete the association authorization after you create the association. This step prevents you from recreating the same association later. To delete the authorization, reconnect to the EC2 instance in Account A. Then, run the following command:aws route53 delete-vpc-association-authorization --hosted-zone-id <hosted-zone-id>  --vpc VPCRegion=<region>,VPCId=<vpc-id> --region us-east-19.    Complete the following steps to confirm VPC association to the private hosted zone:Open the Route 53 console.Select the private hosted zone domain name.Verify the VPC ID of Account B Hosted zone details.EC2 instances in the VPC from Account B can now resolve records in the private hosted zone in Account A.Related informationWorking with private hosted zonesFollow"


Question: How can I set up Application Load Balancer authentication using Facebook and Amazon Cognito as identity providers?

How can I set up Application Load Balancer authentication using Facebook and Amazon Cognito as identity providers (IdPs)?

Answer: "How can I set up Application Load Balancer authentication using Facebook and Amazon Cognito as identity providers (IdPs)?Short descriptionWith Application Load Balancer authentication, the Application Load Balancer either confirms that the client is authenticated or prompts the client to authenticate. The backend target is responsible only for running other business logic or services, such as user profile and payments. The Application Load Balancer is the gatekeeper for denying and allowing client access. However, because Application Load Balancers don't store client login credentials, you can configure Amazon Cognito and Facebook to manage and authenticate users.ResolutionSet up the Facebook applicationGo to Facebook for developers on the Facebook site.On the top right, choose My Apps, and then choose Add New App.Specify a Display name, and then choose Create App ID.Choose Facebook Login.Choose Web (www).Specify the domain name of the site. This name is the same as the name that you use to alias to the Application Load Balancer's DNS.Choose Next.Skip the JavaScript SDK, and then choose Next.Choose Next.When you get to step 5 in the wizard, continue to set up Amazon Cognito. There's one final step to complete when setting up the Facebook application, but you must set up Amazon Cognito first.Set up the Application Load Balancer for authentication, and then set up Amazon Cognito as an IdP/IdP aggregatorOpen the Amazon Elastic Compute Cloud (Amazon EC2) console.Create an Application Load Balancer.On the navigation pane, under LOAD BALANCING, choose Load Balancers.Select the Application Load Balancer that you created in step 2.Choose Listeners, Add listener.Choose HTTPS:443(or any port with the HTTPS protocol).For Action, choose Authenticate.Keep the Amazon Cognitoselection.For Cognito user pool, choose Create new.For Social IDP, choose Facebook.Go to Facebook for developers on the Facebook site.Select the application that you created in the Set up the Facebook applicationsection of this procedure.Choose Settings, and then choose Basic.Copy the App ID.Choose Show, and then enter your Facebook password to select App Secret.Return to the Amazon Cognito setup page in the Amazon EC2 console. Then, paste the App IDand enter the App Secret.For Authorize scope, enter public_profile.Under Domain prefix, choose a unique name to append to the Amazon Cognito regional DNS service. For example, if you specify "abc" as your domain prefix, your fully qualified domain name (FQDN) is https://abc.auth.us-east-2.amazoncognito.com/.Choose Create Cognito User pool.Copy the domain name you specified in step 18.Add another rule to your HTTPS listener. Choose **Forward to:**and then specify the target group name of your Application Load Balancer.Choose Save.Finish setting up the Facebook applicationOpen the Facebook application.Choose Facebook Login, and then choose Settings.For Valid OAuth Redirect URIs, paste the Amazon Cognito FQDN and add a suffix of /oauth2/idpresponse. For example, https://abc.auth.eu-west-1.amazoncognito.com/oauth2/idpresponse.Choose Save changes.Choose Settings, Basics, App domains.Add the domain name that points to your Application Load Balancer.Choose Save changes.Finish setting up Amazon CognitoOpen the Amazon Cognito console.Choose your configured user pool.Choose App client settings.For Callback URL(s), specify the domain name that you created an alias record for in Route 53 that points to the Application Load Balancer, and then add the /oauth2/idpresponsesuffix. For example, https://www.example.com/oauth2/idpresponse.**Important:**Facebook is a third-party application, which means that the configuration steps above might change over time. For the latest updates, refer to Facebook's documentation on the Facebook site.Related informationAuthenticate users using an Application Load BalancerSimplify login with Application Load Balancer built-in authenticationFacebook (Identity pools)Register with a social IdPFollow"


Question: "I tried to delete my Amazon VPC, but I received a dependency error. How can I delete my Amazon VPC?"

I receive a dependency error when I try to delete my Amazon Virtual Private Cloud (Amazon VPC).

Answer: "I receive a dependency error when I try to delete my Amazon Virtual Private Cloud (Amazon VPC).Short descriptionWhen you delete an Amazon VPC, all its components are also deleted. These components include subnets, security groups, network access control lists (network ACLs), route tables, internet gateways, and DHCP options. However, before you attempt to delete your Amazon VPC, you must first delete or disassociate all dependent resources. Follow these steps in the order listed here to avoid dependency errors.ResolutionBefore you delete your Amazon VPC, you must delete the dependent resources.You can use the AWS Command Line Interface (AWS CLI) delete-vpc command or the Amazon VPC console to delete your Amazon VPC.Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.Resolve errors received when trying to delete an Amazon VPC through the AWS CLIThe following error indicates that there are dependencies that you must remove before you can delete the Amazon VPC:"An error occurred (DependencyViolation) when calling the DeleteVpc operation: The vpc 'vpc-id' has dependencies and cannot be deleted."To resolve this error, complete the following steps:1.    Run the following script to find the remaining dependencies:#!/bin/bashvpc="vpc-xxxxxxxxxxxxx"region="yy-yyyy-y"aws ec2 describe-internet-gateways --region $region --filters 'Name=attachment.vpc-id,Values='$vpc | grep InternetGatewayIdaws ec2 describe-subnets --region $region --filters 'Name=vpc-id,Values='$vpc | grep SubnetIdaws ec2 describe-route-tables --region $region --filters 'Name=vpc-id,Values='$vpc | grep RouteTableIdaws ec2 describe-network-acls --region $region --filters 'Name=vpc-id,Values='$vpc | grep NetworkAclIdaws ec2 describe-vpc-peering-connections --region $region --filters 'Name=requester-vpc-info.vpc-id,Values='$vpc | grep VpcPeeringConnectionIdaws ec2 describe-vpc-endpoints --region $region --filters 'Name=vpc-id,Values='$vpc | grep VpcEndpointIdaws ec2 describe-nat-gateways --region $region --filter 'Name=vpc-id,Values='$vpc | grep NatGatewayIdaws ec2 describe-security-groups --region $region --filters 'Name=vpc-id,Values='$vpc | grep GroupIdaws ec2 describe-instances --region $region --filters 'Name=vpc-id,Values='$vpc | grep InstanceIdaws ec2 describe-vpn-connections --region $region --filters 'Name=vpc-id,Values='$vpc | grep VpnConnectionIdaws ec2 describe-vpn-gateways --region $region --filters 'Name=attachment.vpc-id,Values='$vpc | grep VpnGatewayIdaws ec2 describe-network-interfaces --region $region --filters 'Name=vpc-id,Values='$vpc | grep NetworkInterfaceIdaws ec2 describe-carrier-gateways --region $region --filters Name=vpc-id,Values=$vpc | grep CarrierGatewayIdaws ec2 describe-local-gateway-route-table-vpc-associations --region $region --filters Name=vpc-id,Values=$vpc | grep LocalGatewayRouteTableVpcAssociationIdNote: In the preceding script, enter your VPC ID in the vpc field and AWS Region (for example, us-east-1) in the Region field. Also, make sure that you're using credentials with appropriate permissions to run the Describe API calls.2.    Delete the remaining dependencies that you identified in step 1, and then retry deleting your Amazon VPC.Resolve errors received when trying to delete an Amazon VPC through the Amazon VPC console"VPC contains one or more instances, and cannot be deleted until those instances have been terminated."This error indicates that the Amazon VPC has Amazon Elastic Compute Cloud (Amazon EC2) instances running on it. To resolve this error, terminate your instances."VPC contains one or more in-use network interfaces, and cannot be deleted until those network interfaces have been deleted."This error indicates that the Amazon VPC has network interfaces that are in use or available."An error occurred (InvalidParameterValue) when calling the DeleteNetworkInterface operation: Network interface 'eni-aabbccdd' is currently in use."This error indicates that there are requester-managed network interfaces that you can't delete. To delete requester-managed network interfaces, you must delete the AWS service that created the network interfaces.Follow these steps to remove the dependent services:1.    Open the Amazon EC2 console.2.    Select the AWS Region that the Amazon VPC is in.3.    In the navigation pane, under Network Interfaces, search for the VPC ID of the Amazon VPC that you're deleting.4.    Select the network interface and choose the Details tab.5.    Review the Description to see which resources the network interface is attached to.6.    Delete the associated resources. For example, you're deleting a Network Interface and the Requester ID is amazon-elb. Use the value in the Description field of the Elastic Network Interface to identify the load balancer. Then, navigate to the Load Balancer section of the Amazon EC2 console, locate the load balancer, and delete it. Note: If the network interface is a primary one, then it's deleted when you delete the instance."VPC has one or more attached NAT gateways or virtual private gateways, and cannot be deleted until they are detached or deleted."This error indicates that the Amazon VPC has a dependency. To delete them, follow the steps in Delete a NAT gateway and Detaching and deleting a virtual private gateway."The VPC is peered as a requester with at least one other VPC through peering connections."This error indicates that there are active VPC peering connections in the Amazon VPC. To delete them, follow the steps in Delete a VPC peering connection.If you still have dependencies that are blocking the Amazon VPC from deletion, then use the following procedures.To delete the carrier gateway:1.    Open the Amazon VPC console.2.    In the navigation pane, choose Carrier Gateways.3.    Select the carrier gateway, and then choose Actions. Then, choose Delete carrier gateway.4.    In the Delete carrier gateway dialog box, enter Delete, and then choose Delete.To disassociate the VPC from any local gateway route tables:1.    Open the AWS Outposts console.2.    In the navigation pane, choose Local gateway route tables.3.    Select the route table.4.    Choose Actions, and then choose View details.5.    In VPC associations, select the VPC to disassociate, and then choose Disassociate.6.    Choose Disassociate.Related informationWhy can't I detach or delete an elastic network interface that AWS Lambda created?How can I delete my VPC that is shared with another AWS account?Why can't I delete my requester-managed VPC endpoint?Follow"


Question: How can I use Data Pipeline to copy tables from one database table to another?

"I want to use AWS Data Pipeline to copy a table to a different database. For example, I want to copy an Amazon Relational Database Service (Amazon RDS) for MySQL table or Amazon RDS for PostgreSQL table to Amazon Redshift."

Answer: "I want to use AWS Data Pipeline to copy a table to a different database. For example, I want to copy an Amazon Relational Database Service (Amazon RDS) for MySQL table or Amazon RDS for PostgreSQL table to Amazon Redshift.ResolutionCopy an Amazon RDS for MySQL table to Amazon RedshiftCreate a pipeline using one of the following Data Pipeline templates:Full copy of Amazon RDS for MySQL table to Amazon RedshiftIncremental copy of an Amazon RDS for MySQL table to Amazon RedshiftFor more information about these templates, see Amazon RDS to Amazon Redshift templates.Copy an Amazon RDS for PostgreSQL table to an Amazon Redshift tableUse the PostgresqlRdsToRedshift scripts from the AWSLabs GitHub repository. Before you begin, modify the postgresqlRDS-to-Redshift-definition.json file with the following:Your databaseYour Amazon Elastic Compute Cloud (Amazon EC2) instanceThe path to your hosted shell script in Amazon Simple Storage Service (Amazon S3)Note: Be sure to modify the JSON file according to your use case.Copy an Amazon RDS for MySQL table to an Amazon RDS for PostgreSQL tableUse the MySqlRdsToPostgreSqlRds scripts from the AWSLabs GitHub repository. Before you begin, modify the mysqlRDS-psqlRDS-copy-using-shell-definition.json file with the following:Your databaseYour EC2 instanceThe path to your hosted shell script in Amazon S3Note: Be sure to modify the JSON file according to your use case.Related informationDefine a pipeline in JSON formatFollow"