Great post from our friends at Source link
A drawing should have no unnecessary lines and a machine no unnecessary parts.
In the book Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann has written about traits and trade-offs for data infrastructure while designing modern applications. He has given an example architecture for a data system that combines several components. I used this example for the article Example Architectures for Data-Intensive Applications. That article explored just the Couchbase features and functions.
A recent Twitter thread talked about how AWS has many databases, but most “half-baked” services (not my characterization) compete for the same business as DynamoDB! Could this be a paradox of choice? The theory is, more choices do not usually lead to better outcomes. However, customers do vote with their dollars and cents. So far, they’re picking AWS for the potential savings in the short run. Could this lead to complexity and drive up the cost in the future?
In this article, I explore how we can architect that example using products and components from various Cloud database vendors.
- Amazon AWS
- Microsoft Azure
- Google GCP
Example Reference Architecture
Amazon has choices and a Reddit user has created a decision tree for choosing the databases! Here’s that graphic.
Using this guide, let’s look at the architecture to make our choices for various components.
For the core operational database, you can use DynamoDB (NoSQL), DocumentDB(NoSQL) or Aurora/RDS (RDBMS). Add either Elasticache or MemoryDB for caching. Use Amazon Neptune or Elastic Search service (using OpenSearch). Or use Elastic itself directly. Either AWS SQS or Kafka provides queuing function for downstream processing. Amazon Redshift and Amazon Quicksight provide the data warehouse and BI function.
You’ll have to make choices about sizing and workload about each one of them and find a way to stitch them together either by built-in functionality, connectors, and customer application code. The lines from the reference architecture diagram are intact. Choice does address your need even if you have to stitch them together.
Microsoft has grown from a single (Sybase) SQL Server on OS/2 in the 90s plethora of products and services on Azure. This page gives you the Azure databases list and matches them with the right workload requirements. Just like AWS, the choices are plenty. Interestingly, they don’t list the “search” in the database category. This is typical of database vendors even though most applications that use databases require search. Microsoft has the Azure cognitive search service, not tightly integrated with Cosmos or other services. You also have the option of deploying Elasticsearch itself.
In the end, this is the same as the AWS diagram above, replaced with Microsoft Azure products.
Similar to AWS and Azure, GCP has plenty of choices for databases from Cloud SQL to Cloud Spanner, Memorystore to Firestore, and Oracle to Bigtable.
Different Cloud, same result. You can pick up from a number of databases, components for cache, queues, and search from GCP and others. The architecture is the same and the challenges are the same you saw with AWS native solutions.
MongoDB is a scale-out, highly available documented oriented database that meets a number of requirements for a modern database. MongoDB provides a simple get/set API, complex aggregation framework, and distributed transactions. MongoDB reduces the number of components because it has built-in MongoDB Realm functions which can execute serverside logic. You’d still need a caching layer to meet low latency performance objectives, While the product is advertised to have built-in text search, its functionality is limited. So is its capability for complex analytical queries.
This improves the situation because it has an Atlas search. Atlas search is built using Apache Lucene. This search capability is only available on Atlas and not within the MongoDB product.
Couchbase and Couchbase Capella
Couchbase is a shared-nothing, scale-out, highly available, high-performance document-oriented NoSQL database that also meets a number of requirements for a modern database. Couchbase provides simple get/set API, SQL inspired N1QL for both operational and analytical query processing, full-text search, and event processing. It’s distributed and supports distributed transactions both for document get/set APIs and N1QL and their combinations.
Couchbase FTS is built from the ground up and uses the bleve search engine. Couchbase Analytics is an MPP database engine designed for analytics. And finally, it’s the same core product on-prem and on-cloud.
On-prem and on-cloud, there are plenty of choices for databases to design data-intensive applications. Do each architect and company have different philosophies: Best of the breed or integrated solution? The answer, of course, is “it depends”. Even in this simple example, each database may have optimizations, and would be good to hear from respective experts.
#Sample #Architecture #Amazon #AWS #Microsoft #Azure #Google #GCP #MongoDB #Couchbase