BigQuery is a fully managed, cloud-based data warehousing service provided by Google. It is built on top of the Google Cloud Storage and Google File System (GFS) and uses a distributed architecture to process large amounts of data.
The basic architecture of BigQuery consists of the following components:
User Interface: The user interface is the front-end of BigQuery and provides access to the service through a web console, command-line interface, and APIs.
Query Engine: The query engine is responsible for processing SQL queries and returns results to the user. It is built on top of Dremel, a highly parallel, columnar, and distributed query engine. Dremel supports complex, nested data structures and can perform analytical queries on large datasets in seconds.
Data Storage: BigQuery stores data in a columnar format using Capacitor, a highly-scalable and efficient storage format. The data is stored in Google Cloud Storage and is distributed across multiple nodes in a cluster.
Data Processing: BigQuery uses a MapReduce-like model to process data. The data is split into smaller chunks called “shuffles,” which are processed in parallel by different nodes in the cluster. The results are then combined and returned to the user.
Resource Management: BigQuery uses a shared-nothing architecture, which means that each node in the cluster has its own resources and is responsible for managing them. This allows BigQuery to scale horizontally and handle high concurrency.
BigQuery is optimized for low-latency, high-concurrency, and high-throughput queries. It is able to process petabytes of data in seconds, and it is able to handle high concurrency of concurrent users and jobs.
Read more blogs on bigquery here
BigQuery import urls to refer
