Overview
A ThoughtSpot deployment consists of the following high level software systems:
- Operating System (OS) and software packages installed on the OS
- Third-party software
- ThoughtSpot application services (binaries and configuration)
Operating System
All ThoughtSpot physical appliances, virtual machines (VMs) and public cloud images come pre-installed with CentOS 7. The CentOS distribution of Linux is owned by RedHat and closely tracks versions of RedHat Enterprise Linux (RHEL).
ThoughtSpot uses the minimal install of CentOS 7 with the addition of a few software packages (e.g. Python) needed for ThoughtSpot operations. The most notable change to the installation is to the Linux kernel, which is sourced from the current long term stable kernel version instead of the default included in CentOS 7 (kernel-lt package). To list all the installed packages, see Checking Package Versions.
Third-Party Software (Middleware)
Third party software used includes Java, Boost C++ libraries, Google protocol buffers, etc. These are software components necessary for operation of the ThoughtSpot application. ThoughtSpot only uses software licensed for distribution.
Development and Release Process
ThoughtSpot releases its software as a tarball containing all the ThoughtSpot application (binaries and configuration), third-party software, and an operating system image. Installation or update using this release tarball on appliances, VMs, or cloud instances updates each of these components.
Operating System
Building the operating system image including software packages is a multi-step process:
-
Begin with the set of packages in the base OS image and our added packages.
-
Configure all installation to only use official public RedHat repositories.
-
For each package, install the current stable version including any security patches.
-
Bring up the image on all supported platforms for stability and performance testing along with the ThoughtSpot application stack. Success criteria: no OS impact on stability or performance.
-
Scan the Operating System and ThoughtSpot application stack using Qualys scans with additional modules enabled: Vulnerability Management, Web App Scanning.
-
Review all vulnerabilities found. Success criteria is zero severity 4+ vulnerabilities.
-
Assuming all above testing and exit criteria are met, the OS image is considered qualified.
Third-Party Software
Third-party software is periodically sourced from the upstream distribution of each software component. Unlike OS and ThoughtSpot application, this changes less frequently and on an as needed basis, when any new security vulnerability or stability issue is discovered in the library. The list of all third-party software as well as licensing details are here.
ThoughtSpot Application
ThoughtSpot follows industry standard best practices for writing robust software. Every code change is reviewed by at least one engineer. Our engineering team consists of senior engineers from Enterprise software and web companies.
ThoughtSpot uses a small number of proven programming languages powering some of the largest enterprises in the world. ThoughtSpot tracks stability, performance, and reliability of our software and services aggressively. The ThoughtSpot platform is trusted by dozens of global F2000 organizations.
Protection of Source Code
Source code is private and not shared publicly, e.g. all distribution to customers is in binary or minified format to discourage reverse engineering.
Automated Tools
We use automated tools and infrastructure like Jenkins, Kubernetes, AWS, partnering with the teams behind these systems so as to adopt best practices. For example, all our automation runs through Jenkins, which is managed by CloudBees (the company behind Jenkins) using an enterprise license with regular security patching, and so on. We upgrade our automation tools regularly.
Independent Testing
Independent testing is done outside of the product team by pre sales and post sales before promoting to production. Some areas are tested by third party testers.
Security Hardening
Starting 4.5.1.5, we have also taken specific steps to incorporate most of CIS standard recommendations towards hardening.
Installation and Upgrade Process
ThoughtSpot is installed or updated from a release tarball which contains the ThoughtSpot application (binaries and configuration), third-party software, and Operating System image.
Operating System Image Installation
Installing ThoughtSpot on any node (VM, cloud instance, appliance) automatically updates the operating system and required packages on the node. No Internet or repository access is required for this, the update is applied directly from the release tarball.
Specifically, all nodes running ThoughtSpot are required to have two root partitions on their boot drive of which one of them is booted from at any given time. During installation or update, the Operating System image contained in the release tarball is copied into the second currently-unused root partition and the system switches to it through a reboot.
Checking OS Package Versions
The following command run from any ThoughtSpot node will indicate versions of all installed packages:
rpm -qa
Upgrades
ThoughtSpot patches the Operating System at the time of upgrades. The exact same process used during installation is also applied during upgrades. The previous OS image on a node gets replaced by the new image carried in the release tarball.
Only some releases may patch the Operating System, not all. Typically, all major and minor releases (e.g. 4.4, 4.5, 4.5.1, 5.0) upgrade OS patches, whereas only some patch releases (e.g. 4.4.1.4) contain OS patches.
Distributed Clusters and Failure Handling
On distributed clusters, individual nodes receive the OS image from the release tarball individually. Initially, the new image is deployed on a single node only. When that node is deemed healthy following the update and a rich set of tests, the image is made available to remaining nodes in the cluster.
If a node fails to patch, then ThoughtSpot support will modify the upgrade workflow to either retry the patching or skip and exclude the node.
Third-Party Software
Installation or upgrade of ThoughtSpot deployments automatically upgrades all third-party software to the version included in the release tarball.
Security Scanning and Patching Process
The ThoughtSpot Security team continuously scans security bulletins for new vulnerabilities discovered in included OS packages (e.g., Linux Kernel, libc) and third party software (e.g., Java). Additionally, weekly scans are done for all release branches using Qualys with the following additional modules enabled: Vulnerability Management, Web App Scanning. The security scans discover vulnerabilities at all layers: OS, third-party software, as well as ThoughtSpot application binaries and configuration. Additionally, ThoughtSpot periodically scans all source code for third-party software as well as ThoughtSpot’s proprietary code base for vulnerabilities or unsafe usage using SourceClear.
After a critical new vulnerability is found (severity 4 or 5), ThoughtSpot includes the corresponding patch in the next patch release for all supported release branches. Consult ThoughtSpot documentation or support to find out if you are on an active or supported release branch.
After a new patch release with a critical security vulnerability is available, customers are encouraged to upgrade their deployment quickly.
Latency
We recommend customers to wait for the next regular release for receiving security patches. However, should a critical vulnerability be discovered in the interim, ThoughtSpot can push out a new patch release containing the required patches, if available upstream.
ThoughtSpot targets a three week or less cadence for generating patch releases for all supported release branches. Timeline for the new release and patching depends on availability of the patch upstream (e.g., not all vulnerabilities in Linux are immediately fixed) and qualification (ThoughtSpot qualifies each build on each supported cloud and on-prem platform). If a fix is unavailable upstream at the moment, customers and ThoughtSpot support can work together to identify potential workarounds.
Storage Security
Encryption at Rest
-
On-prem: Not supported yet
-
Cloud: Supported on AWS (See https://aws.amazon.com/), GCP (See https://cloud.google.com/%2F%5D), Azure (See https://azure.microsoft.com/en-us/)
Secure Erase
Current erase guide (See https://thoughtspot.egnyte.com/dl/E1eYDyfotL/SOP-520-0007-00-User-Data-Removal.pdf_)