Node
Overview
Despite being referenced as a "database", a Kwil Database contains much more than just a simple database engine. On each node, there are three to four separate processes running that enable the functionality of a Kwil Database.
Always Present:
Reverse Proxy Server
Kwil DB Synchronizer
Database Engine
Optionally Present:
Advanced Request Gateway
Reverse Proxy Server
The reverse proxy server is the entry-point for all requests sent to a node. While any standard reverse proxy server can be used, we recommend NGINX due its lightweight nature. On a Kwil node, the reverse proxy server serves four main purposes:
Manages TLS/SSL certificates, ensuring encrypted communication with any clients / peer nodes.
Enforces a CORS (Cross-Origin Resource Sharing) configuration, dictating from which sources (both clients and peers) to accept data read/writes.
Routes incoming requests to the proper gateway or synchronizer, and prevents circumvention of any Advanced Request Gateways.
Serves static resources from the local filesystem.
Kwil DB Synchronizer
The Kwil DB Synchronizer is the application that sits directly above the database engine. All incoming requests go through the Kwil DB Synchronizer, which dictates whether or not the request should be processed, whether a data-write should be propagated to the rest of the network, and whether or not an incoming piece of data should be written (and subsequently served) from the local filesystem.
The Kwil DB Synchronizer is also responsible for managing data moat block bundling functionality. Kwil DB Synchronizers have the ability to write multiple data moat blocks to a single Arweave bundle, allowing for practically unlimited scalability and network data-throughput. This system also allows users to cross-pollinate data moats among different nodes, which encourages further data decentralization and replication while keeping the relative operating overhead costs low.
On each request, the Kwil DB Synchronizer decides whether or not the request should be propagated to the network, and in which ways the propagation should occur. In each Kwil DB Synchronizer configuration file, the node operator can decide if they want rapid peer synchronization (and with which peers to synchronize with), to which data moat block the data should be written, and the frequency in which those data moat blocks are published to Arweave.
Each node continually maintains an open cache for each data moat it is serving. Periodically, the node will go through and seal each cache, while subsequently creating a new cache for the data moat. The node automatically identifies empty caches, and if detected, will discard the sealed cache and stop maintaining that moat's cache until it receives a new data entry for that moat. Once a cache is sealed, the node will aggregate all data into a single bundle, sorted by the moat and data type (this process is internally called "finalizing"). Then, each bundle will be submitted to Arweave and moved into a "pending" pool. Periodically, the node will check each bundle to see if it has been mined on a block on Arweave. If the bundle has been dropped from the network, it will resubmit. If the bundle is still waiting to be mined but is active, the node will keep the bundle in the pending pool. If the bundle has been mined, the node will delete the bundle from the pending pool, and subsequently register the bundle in a block index (if necessary). At this point, the bundle has reached a level of data permanence, and has also been made possible for discovery by other nodes.
Database Engine
At the heart of each node is the database engine. By default, Kwil nodes are equipped with a standard relational database (Postgresql), however virtually any type of database engine (even non-relational) can be used.
Each database engine is configured to only allow access via the Kwil DB Synchronizer that is running on the same machine. This is done for two reasons:
The chance for potential user error is significantly reduced by restricting database access to be forced through the Kwil DB Synchronizer. Non-propagating data writes must be explicitly identified to the Kwil DB Synchronizer, instead of accidentally being submitted to the database engine. This makes it more unlikely that similar nodes in a network contain differing data.
By restricting access to the Kwil DB Synchronizer, which is further restricted by the reverse-proxy server (and optionally the advanced request gateway), database users enjoy a significantly reduced attack vector, reducing the chance of network abuse by bots, data / privacy breaches, and cross-node state divergence.
Advanced Request Gateway
Kwil Nodes can be optionally equipped with an Advanced Request Gateway. At its core, an Advanced Request Gateway is an application middleware that can contain complex business logic for defining database interactions. This ultimately extends the functionality of a Kwil Node from just a database/filesystem, to a full-featured decentralized API system. Examples of what can be built as an Advanced Request Gateway include: fine-grained IAM database access, complex multi-query API requests, and rules for what data gets passed to the Kwil DB Synchronizer for propagation.
All Kwil Social pages run on our own custom Advanced Request Gateway, which manages application specific logic. An example of this is in the use of our "Groups" functionality, in which a Group's data-write access is managed by checking an RSA signature against a username's public modulus. The benefit of a system such as this is that it only requires one gateway to verify access rights, but all peer nodes have the option to check via the RSA signature to independently verify whether or not each user has the permissions to write data.
Last updated