Designing Data-Intensive Applications

Who should listen to this book? If you develop applications that have some kind of server or back-end for storing or processing data, and your applications use the internet, for example web applications, mobile apps, or internet-connected sensors, then this book is for you. This book is for software engineers, software architects, and technical managers who love to code. It is especially relevant if you need to make decisions about the architecture of the systems you work on, for example if you need to choose tools for solving a given problem and figure out how best to apply them. But even if you have no choice over your tools, this book will help you better understand their strengths and weaknesses. You should have some experience building web-based applications or network services, and you should be familiar with relational databases and SQL. Any non-relational databases and other data-related tools you know are a bonus, but not required. A general understanding of common network protocols like TCP and HTTP is helpful. Your choice of programming language or framework makes no difference for this book. In essence, you'll find this book valuable if you want to learn how to make data systems scalable, for example to support web or mobile apps with millions of users. You need to make applications highly available, by minimizing downtime, and operationally robust. You are looking for ways of making systems easier to maintain in the long run, even as they grow and as requirements and technologies change. Or you have a natural curiosity for the way things work and want to know what goes on inside major websites and online services. This book breaks down the internals of various databases and data processing systems, and it's great fun to explore the bright thinking that went into their design. Sometimes, when discussing scalable data systems, people make comments along the lines of, You're not Google or Amazon. Stop worrying about scale and just use a relational database. There is truth in that statement. Looking for scale that you don't need is wasted effort and may lock you into an inflexible design. In effect, it is a form of premature optimization. However, it's also important to choose the right tools for the job, and different technologies each have their own strengths and weaknesses. As we shall see, relational databases are important, but not the final word on dealing with data. Scope of this book This book does not attempt to give detailed instructions on how to install or use specific software packages or APIs, since there is already plenty of documentation for those things. Instead, we discuss the various principles and trade-offs that are fundamental to data systems, and we explore the different design decisions taken by different products. We look primarily at the architecture of data systems and the ways they are integrated into data-intensive applications. Topics such as deployment, operations, security, and management, among others, are complex and important, and we wouldn't do them justice by making them superficial side notes in this book. They deserve books of their own. Many of the technologies described in this book fall within the realm of the big data buzzword. However, the term big data is so overused and underdefined that it is not useful in a serious engineering discussion. This book uses less ambiguous terms, such as single-node versus distributed systems, or interactive versus batch processing systems. This book has a bias toward free and open-source software, because reading, modifying, and executing source code is a great way to understand how something works in detail. Open platforms also reduce the risk of vendor lock-in. However, where appropriate, we also discuss proprietary software, closed-source software, software-as-a-service, or companies' in-house software that is only described in literature but not released publicly.