GCP Log Discovery and Collection in USM Anywhere

The Google Cloud Platform (GCP) Sensor uses the Google Cloud Pub/Sub to power asynchronous log collection. Cloud Pub/Sub operates based on an export/subscribe model wherein your environment's logs are exported to one or more topics via export sinks. A Pub/Sub subscription, listening to that topic, will deliver the collected logs to your GCP Sensor. Refer to the Google Cloud Pub/Sub documentation for a more in-depth understanding of how this operates.

Complete the following steps before enabling Cloud Pub/Sub:

  • Create a Cloud Pub/Sub topic that will receive your exported logs, and a subscription for that topic
  • Create one or more export sinks to define which logs your GCP Sensor will receive
  • Create filters to determine which logs are collected by each of your export sinks

Warning: You must have the Cloud Pub/Sub API enabled before beginning these steps.

Create a Topic

In Google Cloud Pub/Sub, a topic receives the logs your GCP environment exports. Those logs are then retrieved by your GCP Sensor via subscriptions. Depending on the needs of your particular implementation, you may only need to create a single topic to receive all of your exported logs from all of the export sinks you configure. However, you may find that it would be advantageous for your implementation to include multiple topics, in which case any number of topics are supported.

To create a Cloud Pub/Sub topic

  1. Log into your GCP environment and go to the Topics page under Pub/Sub.
  2. Click Create Topic.
  3. Give this topic a name.
  4. Note: Make note of this name, as you will need to reference it when creating your export sinks.

  5. Under Encryption, be sure that Google-managed key is selected.
  6. Click Create Topic.
  7. Now you are ready to create a subscription for this topc.

To create a subscription for a Cloud Pub/Sub topic

  1. Go to the Topics page under Pub/Sub and open the topic you want to create a subscription for.
  2. Scroll to the bottom of the page and click Create Subscription.
  3. Give your subscription a name using the Subscription ID field.
  4. Note: This is the name that will appear in the UI of your GCP Sensor under the Log Subscriptions tab.

  5. Enter the required information and settings for your subscription.
    1. Delivery type: Select Pull
    2. Subscription expiration: Select Never expire
    3. Acknowledgement deadline: You may leave this as its default, or adjust its value to suit your needs
    4. Message retention duration: You may leave this as its default, or adjust its value to suit your needs
  6. Click Create Subscription.
  7. At this point, you may go to the Sensor Details within your USM Anywhere Sensor and check the Log Subscriptions tab to verify that this subscription appears as expected.

  8. Enable this subscription by clicking Enable.
  9. Important: While your subscription will be visible at this point, it will not begin reporting events until you have configured at least one export sink to publish to this topic.

Create an Export Sink

The export sink is what defines which logs are exported to a particular topic. You can create a single sink to export all of the logs you want your sensor to receive, or create any number of individual sinks to group your exported logs by type, to maximize performance, or for any other reason that suits your specific implementation.

To create an export sink for a project or organization

  1. Log in to your GCP environment and go to the organization or project for which you want to create this sink.
  2. Go to the Exports page under Logging.
  3. Click Create Export.
  4. Enter the following information:
    • Sink Name: An identifiable name for this export sink
    • Sink Service: Using the drop-down list, select Cloud Pub/Sub
    • Sink Destination: Using the drop-down list, select the topic you created for this sink
    • Note: If you haven't yet created a topic for this sink, you can select Create New Topic to create one from this page and immediately use it for your sink. If you do so, you must remember to go to that topic and create a subscription for it or your sensor will not receive any logs from it.

  5. Configure a filter for this sink, following the guidelines in Configuring Export Sink Filters.
  6. Click Create Export.

Important: If your sink and topic are in different GCP projects, or if you are exporting organization-level logs to a Cloud Pub/Sub topic in a project, you must complete some additional steps. See the following sections for detailed instructions regarding those two cases.

To create a sink that will publish to a Cloud Pub/Sub topic in a different project

Note: If you have not already granted your service account permission to this second project, use the instructions in Preparing Your GCP Environment for Sensor Deployment to grant permission to this project now. Be sure to restart the sensor app before proceeding on to step one.

  1. Log into your GCP environment and go to the project for which you want to create this sink.
  2. Go to the Exports page under Logging.
  3. Click Create Export.
  4. Enter the following information:
    • Sink Name: An identifiable name for this export sink
    • Sink Service: Using the drop-down list, select Cloud Pub/Sub
    • Sink Destination: Using the drop-down list, select Use a Cloud Pub/Sub topic in another project
  5. When you make your selection in Sink Destination, the menu item will transform into a text field. Use that field to enter the following, substituting your relevant information where there are variables:

    pubsub.googleapis.com/projects/<project-id>/topics/<topic_name>

    Where the <project-id> you reference is the project your topic resides in.

  6. Configure a filter for this sink, following the guidelines in Configuring Export Sink Filters.
  7. Click Create Export.

To create a sink to publish from an organization to a topic in a project

Important: Unlike the previous methods, it is not possible to use the web user interface (UI) to create an export sink to publish from the organization level to a topic at the project level, so this must be achieved by using the Cloudshell Editor native to your GCP environment to enter the following commands.

Access the Cloud Shell Editor in your GCP environment by clicking on the Activate Cloud Shell button. This will open a new window at the bottom of your screen, which may take a few minutes to finish loading. You will use Cloud Shell to enter the following commands.

  1. Use the following commands to create a filter string:
  2. Note: Use the OR operator to concatenate these strings into one filter that gathers all of the logs you require.

    Cloud Audit Logs

    logName=("projects/<project-id>/logs/cloudaudit.googleapis.com%2Factivity" OR "projects/<project-id>/logs/cloudaudit.googleapis.com%2Fdata_access" OR "projects/<project-id>/logs/cloudaudit.googleapis.com%2Fsystem_event")

    VPC Flow Logs

    logName="projects/<project-id>/logs/compute.googleapis.com%2Fvpc_flows"

    Firewall Logs

    logName="projects/<project-id>/logs/compute.googleapis.com%2Ffirewall"

    Syslogs

    logName="projects/<project-id>/logs/syslog"

  3. Use the following command to create a new sink for your organization:
  4. gcloud logging sinks create \ <sink-name> \ --organization=<organization-id> \ --include-children \ pubsub.googleapis.com/projects/<project-name>/topics/<topic-name> \ --log-filter "logName=(\"organizations/<organization-id>/logs/cloudaudit.googleapis.com%2Factivity\" OR \"organizations/<organization-id>/logs/cloudaudit.googleapis.com%2Fdata_access\" OR \"organizations/<organization-id>/logs/cloudaudit.googleapis.com%2Fsystem_event\")"

    This will return the following message. Make note of the service account name (highlighted here in bold) to enter in the next step.

    Created [https://logging.googleapis.com/v2/organizations/<organization_id/sinks/<sink_name>]. Please remember to grant `serviceAccount:<name-of-sensor-service-account>@<name-of-project>.iam.gserviceaccount.com.com` the Pub/Sub Publisher role on the topic. More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export

  5. Use the following command to grant the service account the permissions it requires:
  6. gcloud organizations add-iam-policy-binding <organization_id> \ --member=<name-of-sensor-service-account>@<name-of-project>.iam.gserviceaccount.com> \ --role=roles/pubsub.publisher

Configuring Export Sink Filters

The filter configured for your export sink determines which logs that sink exports to your topic.

To configure the filters for your sink

  1. Go to the export sink you wish to create a filter for, either when you have first created it or by opening it for editing.
  2. Click the carrot in the text box of your export filter and select Convert to advanced filter.
  3. Use the specifications described in the table below to define which filters will be exported by this sink, separating each filter specification with "OR" (as seen in the image above).

Note: Any logs included in this filter not supported by the GCP Sensor will be discarded by the sensor.

Log Type Filter to Capture This Log Notes
Audit Logs at the Organization Level organizations/<organization-id>/logs/cloudaudit.googleapis.com

To filter these logs further, append:

  • %2Factivity: For activity logs
  • %2Fdata_access: For data access logs
  • %2Fsystem_event: For system events

Audit Logs at the Project Level

projects/<project-id>/logs/cloudaudit.googleapis.com

To filter these logs futher, append:

  • %2Factivity: For activity logs
  • %2Fdata_access: For data access logs
  • %2Fsystem_event: For system events
VPC Flow Logs projects/<project-id>/logs/compute.googleapis.com%2Fvpc_flows  
Firewall Logs projects/<project-id>/logs/compute.googleapis.com%2Ffirewall  
Syslog projects/<project-id>/logs/syslog These logs are delivered via the Stackdriver logging agent