Skip to main content

Private Cloud Setup

When you create an account, you choose whether you want to use the SparkLogs cloud or have a private cloud deployment that uses your own Google Cloud account and resources. Each has its advantages.

A private cloud deployment is serverless and requires zero ongoing maintenance and care. Data ingestion, extraction, and query acceleration are still handled in the SparkLogs cloud.

In a private cloud setup, you authorize the SparkLogs cloud to store and process data in your own Google Cloud account and project. No data is persisted outside of your private GCP project.

This gives you complete control and access to your ingested data, and allows you to meet any unique requirements of your organization (e.g., mandated use of customer-managed encryption keys).

There are two types of private cloud service plans:

  1. Private Cloud ($0.22/GB-ingested, $440/month minimum): You store the data in your own Google Cloud account in your own BigQuery dataset; you do not pay for query costs.
  2. Private Cloud Self-Hosted Querying ($0.10/GB-ingested, $2,000/month minimum): You store the data in your own Google Cloud account in your own BigQuery dataset and you also pay Google for query costs.

Setup Overview​

  1. Have a Google Cloud account setup with billing activated and with rights to grant IAM roles
  2. Prepare the appropriate Google Cloud project
  3. Create a BigQuery dataset
  4. Create a Google Cloud Storage bucket
  5. Grant our service accounts least-privilege access to the dataset, project, and bucket
  6. Use the app to create a SparkLogs workspace
  7. Label the dataset and create a claim folder in the bucket, then verify and provision the workspace in the app

If you plan to retain more than 100 TB of ingested log data, see this additional requirement.

danger

When using the Self-Hosted Querying service plan, it is mandatory to use a BigQuery Enterprise reservation that auto-scales from zero slots. A BigQuery long-term commitment is NOT required, and your auto-scaling baseline slots should be 0 and your max slots should be at least 500. You should also turn on BigQuery Fluid Scaling. With Fluid Scaling, you are only billed by the second for query costs, so you can instantly scale from zero to hundreds (or thousands) of compute slots, and you're only billed for the precise number of seconds when queries are executing. In combination with the unique massive-scale adaptive querying technology in SparkLogs, this ensures that generally the cost of each query is a few pennies.

On-demand query mode in BigQuery is not cost-effective, and SparkLogs will not provision the workspace if you do not have an auto-scaling reservation assigned to your host project.

If you are not using Self-Hosted Querying, then a BigQuery Enterprise reservation is not required, but is highly recommended. When you enable BigQuery Fluid Scaling and setup a reservation that has 0 baseline slots, your costs will always be lower.

If you still choose not to create a BigQuery Enterprise reservation, then to avoid unintentional large costs on querying your own BigQuery data, make sure to set a quota for query usage per day. If you only plan on querying your data from within SparkLogs, we recommend setting the quota to 0 or 1 TB to ensure you do not accidentally incur additional costs by manually querying your data directly. This quota will not affect SparkLogs's ability to query your data, as the query costs that are incurred for queries performed within the product are not charged to your Google Cloud account.

If you do plan on querying your own data directly or if you are using Self-Hosted Querying, then make sure to assign a BigQuery Enterprise reservation that auto-scales from 0 slots to at least 500 slots and enable BigQuery Fluid Scaling.

Detailed instructions​

Prerequisites​

You will need an active Google Cloud account that has an active billing account.

tip

Make sure that the GCP project that you intend to use for SparkLogs resources has billing enabled. Google BigQuery requires that a project has billing enabled. Workspace provisioning will fail if your Google project does not have billing enabled.

If your GCP account has an organization, and if your organization has enabled Domain Restricted Sharing (on by default for orgs created after May 3, 2024), then you will need to add our organization to your policy:

Add our organization ID to the allowed list in your organization policy

Follow the instructions to set the organization domain restricted sharing policy to add our organization ID C00pfa94t to allowed organizations.

danger

Do not remove your own organization from the list of allowed organizations. If you remove your own organization ID from the policy, you can lock yourself out of your Google Cloud organization.

In summary, use the Google Cloud console for Organization Policies to edit the constraints/iam.allowedPolicyMemberDomains constraint and add our organization ID C00pfa94t to the list of allowed organizations.

Prepare the appropriate Google Cloud project​

As you desire, you can use an existing Google Cloud project or create a new one just for SparkLogs. If you use an existing project, the roles below give SparkLogs access to the dataset and bucket you create for it.

Make sure that BigQuery is activated for your chosen GCP project by visiting the BigQuery console.

Create the BigQuery dataset​

SparkLogs stores your log data in a BigQuery dataset that you create.

  1. In the BigQuery console, create a dataset in your project.
  2. Set its location to the region of your SparkLogs workspace. The app refuses a dataset in a different location.
  3. Under advanced options, choose the Physical storage billing model. Physical billing charges for compressed bytes, which keeps log storage costs low.
  4. To use customer-managed encryption keys, set them on the dataset now.

You add the workspace's claim label to this dataset after you create the workspace in the app.

Create a Google Cloud Storage bucket​

A Google Cloud Storage bucket is used to store an archived copy of all of your SparkLogs data, updated daily.

Bucket setup instructions:

  • The bucket must be on the same continent as the region of your SparkLogs workspace.
  • Set the default storage class to Autoclass, or set the default storage class to Standard and then create an Object Lifecycle Management rule to transition data older than 14 days to Nearline or Coldline (depending upon how long you want to retain the data and how often you plan to read any archived data). The application may read archived data if you use the replication feature to copy the archived data to other destinations, but will not otherwise access archived data.
  • We strongly recommend enabling Soft Delete on the bucket to protect against accidental deletions.
  • Based on your compliance requirements, you may want to enable Bucket Lock and Object Versioning. This can help address WORM (Write Once Read Many) and other data retention and compliance requirements, such as those associated with HIPAA, FINRA, SEC, or CFTC.

SparkLogs adds the archive copy of your data to your bucket on a daily basis. You determine how long you want to retain this archived data in your bucket by setting the appropriate lifecycle management policy on your bucket. If you do not want to retain archived data indefinitely, then create a lifecycle rule to delete objects in the bucket after your desired retention period (e.g., 1 or 7 years).

Archived data is a ready-to-go data lake: data is stored in compressed HIVE-partitioned parquet format and can thus be directly queried through any platform that supports this format (AWS Athena or Redshift Spectrum, BigQuery, Azure Synapse Analytics, Apache Spark, and many more).

Grant our service accounts access​

In the BigQuery console, open your dataset, click Sharing, then Permissions. Grant the roles/bigquery.dataEditor (BigQuery Data Editor) role on the dataset to the SparkLogs service accounts:

run-ingestor@itl-p-app-core-base-i9gm.iam.gserviceaccount.com
run-query-api@itl-p-app-core-base-i9gm.iam.gserviceaccount.com
run-backend@itl-p-app-backend-n3cv.iam.gserviceaccount.com

Use the IAM permissions editor for your project to grant the same service accounts the roles/bigquery.user (BigQuery User) role. On datasets the service accounts did not create, this role only lets them list tables and read metadata. See Google's role list for details.

Use the Cloud Storage bucket explorer console to open your bucket, then go to the Permissions tab, and add the roles/storage.objectUser (Storage Object User) role to the following account:

run-backend@itl-p-app-backend-n3cv.iam.gserviceaccount.com

This role lets SparkLogs list, write, and delete objects in the bucket.

Use the app to create a private cloud workspace​

  1. Open the app and go to the create workspace page (this is where you automatically land as a new user).
  2. Choose the Private Cloud (GCP) or Private GCP Self-Hosted Querying button, and fill in the Google Cloud project ID, BigQuery dataset name, archive bucket, and optional archive bucket path prefix.
  3. Click Test Connection to Google Cloud to check access to BigQuery, and correct any errors it reports. The bucket is checked in a later step.
  4. Finish creating the workspace.

The app refuses:

  • A Google Cloud project or bucket that belongs to SparkLogs.
  • A dataset that another SparkLogs workspace uses.
  • An archive bucket path that another workspace uses, or a folder above or below it.
  • A prefix with an empty, . or .. folder name, or with characters Cloud Storage cannot store. Spaces and leading or trailing slashes are removed from the prefix.

Claim the dataset and bucket, then provision​

After you create the workspace, the app shows a setup panel with a claim token for the workspace. The dataset label and the claim folder show SparkLogs that these resources are yours. The panel shows each value with a copy button.

  1. Label the dataset. In the BigQuery console, open the dataset, edit its details, and add a label with the key sparklogs-workspace-claim and the claim token as its value.
  2. Create the claim folder in the bucket. In the bucket, create a folder at the path the app shows: <prefix>/sparklogs-claim-<token>. With no prefix, the folder goes at the bucket root. In a bucket with hierarchical namespace enabled, upload an empty file with that name instead.
  3. Click Verify and Provision. The app checks for the claim folder, writes and deletes a test object in the prefix folder, then checks the dataset label and location and provisions the workspace. If provisioning fails after the bucket is verified, fix the reported problem and click Retry Provisioning.

Cost considerations​

You will pay us a fee of $0.22/GB-ingested (Private Cloud) or $0.10/GB-ingested (Self-Hosted Querying), which covers use of our platform for unlimited users. This includes our cloud receiving and processing ingested log data (including AutoExtract technology) and storing it to your private Google Cloud project.

You will also pay directly to Google Cloud the cost of BigQuery resources used by your SparkLogs account. For example, here is Google Cloud BigQuery pricing for the US region:

  • BigQuery Storage Write API: $0.025/GB-ingested
  • BigQuery Storage: with the Physical storage billing model you choose when you create the dataset, you pay for compressed storage: $0.04/GB/mo or $0.02/GB/mo (data ingested more than 90 days ago). Log data is highly compressible, and usually compresses at 10:1 or better, so storage costs are usually minimal.
  • BigQuery Compute (only for Self-Hosted Querying): With the Private Cloud plan, you will NOT pay for any query costs, as the cost of the queries are covered by SparkLogs. With the Self-Hosted Querying plan, SparkLogs's query acceleration and optimization technologies are included for unlimited queries, and then you will also pay Google for BigQuery compute costs ($0.06/slot-hour). Combining SparkLogs's massive-scale adaptive querying technology with BigQuery auto-scaling from zero slots that instantly scales up/down in one-second increments means that most queries cost just pennies, regardless of how much data is queried. Make sure you enable BigQuery Fluid Scaling.
tip

If you are NOT using the Self-Hosted Querying plan, then your use of SparkLogs will NOT incur BigQuery Compute costs for query processing. In this case, SparkLogs is paying for all query costs for queries initiated from within SparkLogs. If you query your own data directly from within Google Cloud using BigQuery SQL, then you will pay Google Cloud for the normal compute costs associated with those queries. In this case we strongly recommend setting up a BigQuery Reservation Enterprise Edition with auto-scaling (you can scale from 0) for optimal cost savings. Always enable BigQuery Fluid Scaling. If you do not setup an Enterprise reservation, then we strongly recommend setting up a Google Cloud quota for BigQuery query usage per day.

Your Google Cloud usage will scale from zero: your Google cloud costs will be directly proportional to the amount of data ingested and retained. You can take advantage of Google Cloud budget alerts to monitor your Google Cloud costs.

If you prefer simple all-inclusive pricing, you can choose to create a workspace that uses the SparkLogs Cloud instead of a private cloud deployment. Your cost will then be a flat $0.39/GB-ingested (subject to a minimum of $100/month).

Deployments larger than 100 TB​

tip

In a private cloud configuration, if you exceed 100 TB of retained log data, in order to use index-accelerated queries, you will need to subscribe to a Google BigQuery Enterprise or Enterprise Plus reservation and use slots in that reservation for index maintenance (see instructions). You can still use an auto-scaling reservation for this purpose to minimize costs.

This is only required for indexed queries if you exceed 100 TB of retained log data. If you exceed this limit and have not setup a slot reservation for index maintenance, you can still perform queries, and these queries will still benefit from the query-acceleration technology in SparkLogs; however, these queries may take longer or become an adaptive-scale query more often than before.

Advanced configurations​

Customer-managed encryption keys​

Google Customer-Managed Encryption Keys (CMEK) allow you to use your own encryption keys to encrypt your data, rather than Google-managed encryption keys. This allows enterprises to meet unique requirements for data governance, compliance, and privacy.

Customer-managed encryption keys may be stored in software, in hardware (HSM cluster), or even externally outside of Google. See Google security key management for details.

To use CMEK with the data stored by SparkLogs in your GCP project, specify your CMEK configuration when you create the BigQuery dataset.

Once the dataset is created, make sure to set a dataset default key so that any tables created by SparkLogs are using CMEK and your desired encryption key. After your SparkLogs workspace is fully provisioned, you can check if the created tables are using CMEK.

You will pay Google additional usage fees related to Customer-Managed Encryption Keys. SparkLogs does not charge extra for using CMEK.