分布式与存储领域的经典论文清单

对象存储系统

[1] Finding a needle in Haystack: facebook’s photo storage

Facebook 的小文件存储解决方案,也是最经典的图片存储架构。

[2] Windows Azure Storage: a highly available cloud storage service with strong consistency

微软云存储的架构介绍,Stream/Partition 的分层架构,以及同一套架构同时支持了 blob/message/file等多种存储服务。

经典系统设计

[3] The Google File System
[4] Bigtable: A Distributed Storage System for Structured Data

互联网领域分布式存储的开山之作,Google 基础架构的重要组成部分。经典的 master-slaves 架构。

[5] Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications
[6] Amazon dynamoDB: a seamlessly scalable non-relational database service

无中心架构的经典设计案例。

分布式算法基础

[7] Paxos Made Simple
[8] The Raft Consensus Algorithm

分布式一致性算法, Paxos 算法的推理证明过程。Raft 则为工业界多选用的一致性算法。这两个完全看明白,不太容易,有基本的概念与流程就好。

[9] CAP Twelve Years Later: How the “Rules” Have Changed

CAP 理论相关,明白在分布式系统下,一致性(Consistent)、可用性(Available)、分区容灾性(Network partitioning)三者必须要做出取舍,不可能同时满足。

推荐阅读

[10] Megastore: Providing Scalable, Highly Available Storage for Interactive Services
[11] Dremel: Interactive Analysis of Web-Scale Datasets
[12] Large-scale cluster management at Google with Borg
[13] Omega: flexible, scalable schedulers for large compute clusters
[14] F1: A Distributed SQL Database That Scales
[15] Spanner: Google’s Globally-Distributed Database
[16] Spanner: Becoming a SQL System

Google 系列论文

[17] Azure Data Lake Store: A Hyperscale Distributed File Service for Big Data Analytics
[18] The Bw-Tree: A B-tree for New Hardware Platforms
[19] LLAMA: A Cache/Storage Subsystem for Modern Hardware

微软 CosmosDB 的存储引擎所用的索引结构和存储子系统

[20] f4: Facebook’s Warm BLOB Storage System
[21] Cassandra - A Decentralized Structured Storage System

facebook 的相关论文

[22] Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases

AWS Aurora 服务的介绍论文

[23] Frangipani: A Scalable Distributed File System
[24] Petal: Distributed Virtual Disks

Frangipani 是基于 Petal 构建文件系统

[25] RADOS: A Scalable, Reliable Storage Service for Petabyte-scale Storage Clusters
[26] Ceph: A Scalable, High-Performance Distributed File System
[27] CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data

Ceph 系列论文

相关的列表

https://github.com/papers-we-love/papers-we-love/tree/master/datastores
https://github.com/rxin/db-readings
http://css.csail.mit.edu/6.824/2014/schedule.html
http://15721.courses.cs.cmu.edu/spring2016/schedule.html