Core Concepts
- Application: A running software that fulfills product needs.
- Data Plane: Component that includes all the applications and their associated cloud resources (AWS, GCP, Azure, etc).
- Control Plane: Component that manages Data Plane (like making stateful changes to Data Plane), as well as interfacing with customers.
- Observed State: A snapshot of the complete resource and configurations of an application.
- Desired State: A representation of an application’s target state based on customers’ configuration.
- Controller: The only component within Control Plane that makes decisions on Data Plane changes, and actually makes the stateful changes. The Controller reads the Observed State for the application and creates the plans to move the applicatioin into the Desired State incrementally.
- Plan: Proposed changes to an application’s Observed State.
Applications
ML Deployment Velocity
TBD
Scaling
TBD
Resource Management
TBD
- Rebalancing in GFS.
- Placement in K8S.
Reference
- https://aws.amazon.com/builders-library/avoiding-overload-in-distributed-systems-by-putting-the-smaller-service-in-control/
- https://ngrok.com/blog-post/control-plane#in-a-database-as-a-service-system
- https://kubernetes.io/docs/concepts/architecture/control-plane-node-communication/
- https://kubernetes.io/docs/concepts/architecture/
- https://www.splunk.com/en_us/blog/learn/control-plane-vs-data-plane.html
- https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/control-planes
- https://docs.aws.amazon.com/whitepapers/latest/saas-architecture-fundamentals/control-plane-vs.-application-plane.html
- https://docs.aws.amazon.com/whitepapers/latest/saas-architecture-fundamentals/full-stack-silo-and-pool.html
- https://brooker.co.za/blog/2019/03/17/control.html
- https://www.youtube.com/watch?v=O8xLxNje30M
- https://brooker.co.za/blog/2022/07/29/getting-into-tla.html
- https://muratbuffalo.blogspot.com/2023/10/hints-for-distributed-systems-design.html
- https://muratbuffalo.blogspot.com/2011/01/crash-only-software-hotos03.html
- https://www.usenix.org/system/files/nsdi20-paper-brooker.pdf
- https://assets.amazon.science/ee/a4/41ff11374f2f865e5e24de11bd17/resource-management-in-aurora-serverless.pdf
- https://www.youtube.com/watch?v=q4WC_6SzBz4
- https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system
- https://news.ycombinator.com/item?id=32651611
- https://cloud.google.com/files/storage_architecture_and_challenges.pdf
- https://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf