![]() ![]() Turns out these efforts have also helped us improve overall reliability and performance.ĬlickHouse Cloud can be seen as two different independent logical units: Object storage access (especially for writes) is pricey, so we had to look closely at how many files we write, how often, and how to optimize that cost.Object storage latency is slower than local disks, so we had to invest in smart caching, parallel access, and prefetching on top of object store to ensure analytical queries remain fast.We also found based on our benchmarking and experience running real-world workloads that this architecture delivers the most competitive price/performance for the types of analytical workloads we see.Īdditional work resulting from taking this path included:.It greatly simplified data management: no need to size your cluster / storage upfront, no need to physically shard data, no need to rebalance data across nodes as our deployment scales up or down, no compute resources sit idle due to fixed compute / storage ratios present in “shared nothing” architectures.Our reasons for picking this architecture were: The diagram below represents the logical “shared everything” architecture of ClickHouse Cloud. We use object storage (such as Amazon S3) as the primary store for the analytical data, and local disks only for caching, metadata, and temporary storage. Essentially this means that storage and compute are de-coupled and can be scaled separately. We settled on a “shared everything” architecture with “separated storage and compute”. Our goal was to build a cloud offering that any developer or engineer could start using without deep knowledge of analytical databases and without the need to explicitly size and manage infrastructure. At the same time, we made sure we had a direct Slack channel available for each private preview user, so we could hear the voice of the customer directly and address any concerns efficiently. We focused on making the onboarding process as frictionless as possible - system-generated invites, self-service onboarding, and automated support workflows. However, from the start, we put simplicity of use first. We did not charge for this use, as our goal was to learn from seeing real-world workloads, get feedback, and grow with our users. We ran an extensive Private Preview program from May to July, where we invited over 50 prospective customers and partners to use our service. We did not want to build in a silo, so we invited ClickHouse users interested in our offering to join us early to try out the platform. When we had to cut features, we were able to drop what was at the bottom without regrets. We had a strict stack rank of features for each release, and clearly marked blockers vs highly desired and nice-to-have features. backups are tricky at huge data volumes). a fully-functioning Control Plane MVP was built in just a few weeks), and other times, things that seemed simple on paper took a lot longer (e.g. Sometimes we were surprised by how quickly we were able to make progress (e.g. ![]() We invariably planned to do too much in each milestone, and then iteratively re-assessed where we got to and adjusted targets and scope as needed. Our prioritization was driven by our collective experience of building cloud offerings, analysis of the market, and conversations with early cloud prospects about their pain points. ![]() We had to be very judicious about what to prioritize for each milestone, and what types of projects to start in parallel. We decided on milestones in advance – Private Preview in May, Public Beta in October, General Availability in December – and then asked ourselves what was feasible by each of these dates and what resources we would need to get there. So we set an aggressive goal of building this cloud offering in a series of aggressive sprints over the course of a year. ClickHouse has been a very popular open source project since 2016, so when we started the company in 2021, there was significant pent up demand for what we were building. Our timeline and planning process may come across as a bit unconventional. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |