With 3 years of work experience and working on several heavy architectural systems, I have always been fascinated by the beauty of complex system designs. I had the opportunity of working with excellent team members who always made me appreciate the nuances of a software system. While I am just getting started understanding this complex system of software architectures, I felt I might as well keep on adding resources here for my perusal and if anyone else can benefit from it.
Getting started resources -
- David Malan is a Harvard professor, and his CS75 lectures were tremendously helpful to me as I was first starting out. The following lecture covers the fundamentals of scalability, replication, load balancing, and sharding. However, I would recommend to watch his other lectures as well.
- Next, I’ve always considered this book to be an excellent place to begin learning about system design. It will assist you in comprehending the fundamental concepts of distributed systems as well as evaluating various technologies for data processing and storage. It is structured into three sections: Data Systems Foundations, Distributed Data, and Derived Data (Batch and Stream Processing).
- Start with the educative course to learn how to approach high-level system design questions. Grokking the system design series is a nice place to start, although it won’t go into great detail. I also watched Jackson Gabbard’s videos to learn how to talk about any system design issue.
- Donne Martin, a Facebook employee, maintains this github page. It contains practically all subjects and resources related to system design. IT will give you an idea of all you’ll need to know.
Getting into the specifics —
- It’s time to delve deeper into the topics once you have a fundamental understanding of system design. My strategy was to take a problem statement, such as develop an Amazon locker system, and try to solve it. If you get stuck at a certain point in the flow, then try reading up on the solutions.
- To begin, I watched the Narendra L’s youtube videos to gain a better grasp of the various systems. He explores real-world architectures in great detail. I’d suggest first designing and then watching his videos.
- The High Scalability website is another excellent resource. It contains a large number of real-world structures, many of which were authored by the architects themselves. For instance, have a look at the article below from Netflix.
- Following that, you can watch the InfoQ sessions, which have speakers from Instagram, Uber, and Slack presenting the best architectures. There are numerous videos about software architectures on their YouTube channel.
- You may also follow the Exponent channel on YouTube. It features mock system design interviews. Mock interviews were really beneficial to me because they helped me gain confidence and evaluate myself.
Overall, system design is more about practice and familiarity with different architectures. Below are some links that I found helpful during the design process.
Basics :
- https://drive.google.com/file/d/16wtG6ZsThlu_YkloeyX8pp2OEjVebure/view (Exhaustive list that I found on Leetcode)
- http://www.aosabook.org/en/distsys.html
- https://www.hiredintech.com/classrooms/system-design/lesson/53
- https://github.com/shashank88/system_design
- https://medium.com/@i.read.tech.books/system-design-estimating-numbers-e3a0b30f9c19
- https://www.lecloud.net/tagged/scalabilityq
- https://github.com/vasanthk/how-web-works
Databases :
- https://studio3t.com/knowledge-base/articles/nosql-database-types/
- https://hackernoon.com/mongodb-schema-design-86327d8fae83
- https://medium.com/hands-on-apache-hbase/an-introduction-to-apache-hbase-2cdd1d9ff13
- https://blog.cloudera.com/apache-hbase-i-o-hfile/#:~:text=In%20HBase%200.20%2C%20MapFile%20is,kept%20in%20the%20same%20file.
- https://towardsdatascience.com/when-to-use-cassandra-and-when-to-steer-clear-72b7f2cede76
- https://blog.discord.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7
- https://www.scnsoft.com/blog/cassandra-performance
- https://medium.com/@deepshig/the-beauty-of-cassandra-ebff58f37cbc
- https://dzone.com/articles/5-data-storages-for-better-log-management
- https://medium.com/@waleoyediran/spatial-indexing-with-quadtrees-b998ae49336
- https://medium.com/@spybugg/storing-and-retrieving-tree-structures-in-relational-databases-using-python-django-7480f40c24b
- https://www.instaclustr.com/blog/redis-vs-memcached/
- https://medium.com/event-driven-utopia/rise-of-the-streaming-databases-episode-2-apache-pinot-409d4397f74e
Database papers :
- https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf (Bigtable)
- https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf (Dynamo Db)
- https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf (Facebook’s TAO)
- https://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf (Cassandra)
Specific topics :
- https://www.youtube.com/watch?v=JK2MdJAWEGc&t=6546s (Hadoop/Big data)
- https://www.youtube.com/watch?v=zC9cnh8rJd0&t=14442s (Apache Spark)
- https://www.toptal.com/big-data/consistent-hashing#:~:text=Consistent%20Hashing%20is%20a%20distributed,without%20affecting%20the%20overall%20system. (Consistent Hashing)
- http://highscalability.com/blog/2008/7/15/zookeeper-a-reliable-scalable-distributed-coordination-syste.html (Zookeeper)
- https://www.youtube.com/watch?v=-DyWhcX3Dpc&list=PLa7VYi0yPIH2PelhRHoFR5iQgflg-y6JA (Apache Kafka)
- https://www.simplilearn.com/tutorials/big-data-tutorial/apache-strom
- http://ksat.me/a-plain-english-introduction-to-cap-theorem
- https://people.cs.rutgers.edu/~pxk/417/notes/paxos.html
- https://www.youtube.com/watch?v=gS_nHTWZEJ8&t=2648s (Elastic Search)
- http://qnimate.com/overview-of-redis-architecture/
Specific architectures :
- https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c
- https://www.citusdata.com/blog/2017/08/28/five-data-models-for-sharding/
- https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-1-storage-mechanics/
- https://hackernoon.com/part-1-building-a-centralized-logging-application-5a537033da0a
- https://medium.com/walmartglobaltech/an-approach-to-designing-distributed-fault-tolerant-horizontally-scalable-event-scheduler-278c9c380637
- https://www.youtube.com/watch?v=s3GfXTnzG_Y
- https://medium.com/@sandeep4.verma/building-real-time-leaderboard-with-redis-82c98aa47b9f
- https://medium.com/swlh/architecting-a-scalable-notification-service-778c6fb3ac28
- https://hackernoon.com/scaling-websockets-9a31497af051
- https://medium.com/event-driven-utopia/anatomy-of-an-event-streaming-platform-part-1-dc58eb9b2412
- https://medium.com/@morefree7
- https://medium.com/@saisandeepmopuri/system-design-online-judge-with-data-modelling-40cb2b53bfeb
- http://www.ardendertat.com/2012/01/11/implementing-search-engines/
- https://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/
- http://highscalability.com/all-time-favorites/
