Blog

Open-source DSP (Delphix Session Protocol)

Thumbnail
Delphix is pleased to announce a new addition to its open-source software initiative, Delphix Session Protocol (DSP).

Delphix is pleased to announce a new addition to its open-source software initiative, Delphix Session Protocol (DSP).

DSP is a network framework used by the Delphix Engine. It aims to simplify the development of distributed applications by separating the application "business logic" from underlying "network issues." In an earlier blog post, Peng Dai, one of the framework's authors, outlined the DSP architecture and some of its features. In this blog post, I'll share why Delphix built DSP and why it is being open-sourced.

Why DSP? In order to enable data virtualization, Delphix engineers maintain a lot of features requiring reliable and performant communication between hosts. For example, Delphix initial database backups require the reliable transfer of terabytes of data over a WAN. This transfer can take days to complete and the expectation is that the backup should survive a plague of network issues (lossy links, spurious network outages, etc.). Restarting a failed transfer is unreasonably costly and therefore must be avoided. However, Delphix engineers maintaining the feature long to focus on the entire backup process instead of micromanaging the data transfer. DSP allows them to do just that.

By writing their data transfer logic on top of DSP, engineers abstract away typical connection management woes. They also benefit from turn-key features such as encryption, compression and connection trunking that automatically make their feature secure and performant. DSP prevents engineers across Delphix from having to solve the same network programming problems repeatedly; engineers can spend more time designing core application logic and in the end produce a better feature.

Why open-source DSP? Network programming is a challenge faced by engineers external to Delphix as well. We believe DSP can provide value as an open-source project by offering session and presentation layer functionality not fully available in other open-source network frameworks.

Many frameworks, like Netty, ease the development of network-intensive applications but do not abstract away the woes of data serialization or protocol design. Frameworks like Java RMI and Google Protocol Buffers abstract away serialization but do not ease connection management and network error handling. DSP hits the network framework "sweet spot" by offering developers a full package of convenient serialization semantics and robust network abstractions. The collective set of DSP features combine to provide developers with high-level abstractions like asynchronous remote method invocation and end-to-end cancellation. It is through these features that DSP achieves its goal of bringing the local application development experience to the distributed environment.

While DSP was initially created to aid the internal development process, we think it has more far reaching applications. Delphix invites engineers to check out DSP so that the framework may continue to be improved and integrated into other network-intensive applications. Delphix has been excited to engage with open-source software community through Open-ZFS and hopes to see similar engagement with DSP. Checkout the DSP Github page to get started and learn more!