This page provides instructions for troubleshooting common issues found with installing or interacting with the Stackdriver Logging agent.
If installation commands result in errors, then make sure that on Linux you prefix the installation commands with sudo.
If the restart fails, and the log output shows "Disabled via metadata", you are likely running an image from Cloud Marketplace, where the Stackdriver Logging agent is disabled by default. This is controlled by the google-logging-enable instance metadata key (with the value 0). To re-enable the agent, either remove that key or set the value to 1 (see Setting instance metadata).
If the agent is not disabled via metadata, reinstall the agent. See the following section, Reinstalling the agent.
If you see HTTP 429 errors, you might have exceeded your Stackdriver Logging API quotas. You can see your available quota by selecting APIs & services > Dashboard in the GCP Console. Choose the Stackdriver Logging API.
If you see API access or authorization problems, see the following section, Verifying Compute Engine credentials.
If the agent seems to be running normally, but you are not getting data, then you should check that the agent is sending data to the correct project. See the following section, Verifying Compute Engine credentials.
Note: If your VM was launched as part of an App Engine flexible environment or Kubernetes Engine deployment, you do not need to install the Stackdriver Logging agent. Skip this page.
To check that the installation was successful, look for the agent's test log entry in the Logs Viewer.
For Compute Engine VM instances, choose the project that contains the VM instance.
For Amazon EC2 VM instances, choose the AWS LINK project that Stackdriver creates when you connect your AWS account to a Workspace.
Do not choose a Workspace project, unless the project contains your Compute Engine VM instance.
For Compute Engine, choose GCE VM Instance.
For Amazon EC2, choose AWS EC2 Instance.
Select syslog (Linux), fluent.info (Windows), or All logs.
If you see a log entry, "Successfully sent gRPC to Stackdriver Logging API," then the agent installation is complete. This message generated once when the agent is installed and also each the agent is restarted.
For more information about the Logs Viewer, see Viewing Logs.
If you do not see any Running status, then the Logging agent is not running.
If you see that StackdriverLogging is running, then you are running a recent agent version. To determine the specific version, see Getting the version.
If you see that fluentdwinsvc is running, then you should upgrade your agent to the latest version.
New-EventLog   -LogName Application -Source "Test Source"
Write-EventLog -LogName Application -Source "Test Source" -EntryType Information -EventID 1 -Message "Testing 123 Testing."
Do not choose a Workspace project, unless that is the project containing your Compute Engine VM instance.
You should see a log entry with your test message. If so, then the Logging agent is operating correctly.
For a Compute Engine VM instance to run the agent without private-key credentials, the instance must have suitable access scopes and the service account identity being used by the instance must have suitable IAM permissions.
When you create a VM instance, the default scope and service account settings are sufficient to run the agents. Very old instances, or instances for which you have changed the default settings, might not have suitable credentials.
Click the name of your VM instance. The detail page for your instance appears.
Look under the Cloud API access scopes heading on the page.
If you see "This instance has full API access to all Google Cloud Services," then your access scopes are adequate.
If you see next to Stackdriver Logging API that you have Write Only or Full permission, then your instance's access scopes are adequate for the Stackdriver Logging agent.
If you see next to Stackdriver Monitoring API that you have Write Only or Full permission, then your instance's access scopes are adequate for the Stackdriver Monitoring agent.
If you do not have suitable access scopes in your Compute Engine instance, add the needed access scopes to your instance.
Even if your Compute Engine VM instance's access scopes are adequate, your instance's default service account might not provide the right IAM permissions for the agent.
Select View By: Members. You should see a list of people, groups, and service accounts. In the Role column are the roles each member has in your project.
If you see Editor, that role is adequate for all the agents. Editor is the default role assigned to service accounts for Compute Engine.
If you see Logs Writer, that role is sufficient for the Logging agent. For other Logging roles that include the write permission, see Access Control for Stackdriver Logging.
If you see Monitoring Metric Writer, that role is sufficient for the Monitoring Agent. For other Monitoring roles that include the write permission, see Access Control for Stackdriver Monitoring.
If your default service account does not have adequate roles, try editing the roles for your service account in the IAM & admin > IAM page. Add the proper Logging or Monitoring roles to authorize the agent(s): Logging > Logs Writer or Monitoring > Monitoring Metric Writer.
On Compute Engine VM instances, you can configure the agent to use a non-default service account that has the proper authorization. On AWS EC2 VM Instances, you must configure the agent to use such a service account.
The agent looks for an environment variable, GOOGLE_APPLICATION_CREDENTIALS, that holds the name of a file which contains the private-key credentials.
Caution: Use file system or other access control to be sure that the private key is only accessible by the applications that need it on your instances.
Is the private key in place?
Is the private key still valid for the service account?
Does the service account have the roles needed for the agent?
To verify that valid private-key credentials are installed on your VM instance, first verify that the credentials file exists in its expected location, and then verify that the information in the credentials file is valid. Previously-valid credentials can be revoked using the IAM & Admin > Service accounts section of the GCP Console. If valid credentials aren't present, see Adding credentials to replace the existing credentials or to add new ones.
Caution: Other services besides Stackdriver might use private-key credentials on your instance. Replacing existing credentials might prevent other services from working.
If either command displays a file like the one shown below, then your instance might have valid private-key credentials. If both commands display a file, then the file denoted by GOOGLE_APPLICATION_CREDENTIALS is used.
If there are no credential files present, then see Adding credentials.
In the credentials file, project_id is your GCP project, client_email identifies the service account in the project, and private_key_id identifies the private key in the service account. Match this information with what is shown in the IAM & Admin > Service accounts section of the GCP Console.
You are checking a Compute Engine instance, but the GCP project in the credentials file is not the project that contains your instance.
You are checking an Amazon EC2 instance, but the GCP project in the credentials file is not the connector project (named AWS Link...) for your AWS account.
The listed service account doesn't exist. It could have been deleted.
The listed service account doesn't have the right roles enabled: Logs Writer for the Stackdriver Logging agent and Monitoring Metric Writer for the Stackdriver Monitoring agent.
The private key doesn't exist. It could have been revoked.
If the service account is the correct one but the private key has been revoked, then you can create a new private key and copy it to your instance. See Creating service account keys.
Otherwise, you must create a new service account as described in the section Adding credentials.
If you are sure that the problem is not related to credentials, you can skip ahead to Installing on Linux or Installing on Windows.
For a full install of the agent and any needed credentials, see Installing the agent.
The following table lists some common problems that you may encounter with the Stackdriver Logging agent and tells you how to fix them.
On Linux, the Logging agent records errors in /var/log/google-fluentd/google-fluentd.log. The error class Google::APIClient::ClientError indicates there is a problem with permissions or API access.
You may start seeing errors after the agent has been running successfully. For example, someone might have revoked the required permissions from your project or your VM instance.
The agent's installer on Windows fails to run You might have downloaded the installer to a system directory. Move the installer to a non-system directory, such as C:\Users\[USERID]\.
Project has not enabled the API You have not enabled the Stackdriver Logging API in your project. Go to the APIs console and change the status of the Stackdriver Logging API to ON.
Unable to fetch access token (no scopes configured?) Your VM instance does not have suitable credentials. If you are using Amazon EC2, then you must install credentials on your VM instances before installing the agent. See Authorizing the Agent to install credentials.
Authorization failed Your private-key authorization credentials for the Logging agent are not configured correctly. See Verifying private key credentials.
The caller does not have permission The service account used for authorization in your project has insufficient permissions. It might be the default service account used within Compute Engine or App Engine, or it might be a user-defined service account used for private key authorization. The account must have Can edit permission. Change the permission of the service account in your project's IAM page. If necessary, you can modify the access scope for an existing VM using the Changing the service account and access scopes for an instance procedures.
Otherwise, see Authorizing the Agent to fix or replace the credentials.
Logging agent stops ingesting logs in the presence of logrotate The Logging agent may lose track of where it is in the input files when logrotate is set up with the copytruncate setting. It's best to use the nocopytruncate setting to ensure that logrotate moves the files instead of truncating them. If you want to keep the copytruncate setting, the workaround is to restart the agent periodically. Or, you could use the postrotate setting to restart the agent.