Using Python to Build a Customizable Delphix Platform
This past summer, I was placed on the AppData team at Delphix to help build features that allow for customizability with a database platform. Because each platform has its own unique functionality that the Delphix product works with, different engineering teams, such as the SQL Server team or the Oracle Database team, among many others, work on building the support for each.
An AppData toolkit is simply a collection of scripts that get called at multiple points as Delphix works with a database. For example, a script specifying a name and location is called when Delphix attempts to discover a source database. By calling these scripts, toolkits provide a powerful way to control what the Delphix lifecycle will look like for a specific database platform.
Sounds awesome, right? Except for one problem. Toolkits are currently written in a language called Lua. This creates limitations that make writing a toolkit difficult because not many developers know Lua, Lua is not a traditionally object-oriented language (like the way Java or Python is) and lastly, it’s comprised of tiny standard libraries.
Because of these barriers, AppData decided to add support for toolkits written in Python since it’s much more popular and contains extensive standard libraries. It also follows a more object-oriented programming convention with classes and polymorphism that is much friendlier to a toolkit writer than Lua is.
Since AppData’s codebase is written in Java, we needed to implement everything needed to allow Python toolkit scripts to communicate with the existing Java codebase. To allow Python callbacks to Java (and Java callbacks to Python), we used Jython, an open-source Python interpreter. Jython lets embedded Python to run within Java, so a Python script can import and call Java while Java does the same for Python.
In addition to some slight refactoring and funny test cases, using Jython came with a couple quirks that needed to be straightened out:
- Input to a toolkit script was in JSON format, so a translation layer was added to convert JSON to Python. That way, the Python toolkit script could ingest the input.
- Output of a script was in Python, so another translation layer was added to convert Python back into JSON.
- Handling Python errors in Java required writing some wrappers to convert the errors into Java.
Overall, the Python toolkit project didn’t require implementing any particularly tricky algorithms or data structures, but the challenge was structuring all the code to fit within the existing codebase and constantly staying cognizant of the quality of work.
The coolest part about this project was having the opportunity to work on something that has real business impact. As an intern, it’s an amazing experience to be able to build the foundation of a project that will ultimately help Delphix expand to a wider market and get a glimpse of the designs decisions of future iterations of Python toolkits.
Interested in becoming a Delphix intern? Apply today. Or visit our University blog to hear more from interns across our offices in Redwood City, San Francisco, Atlanta, Boston, Colorado, UK, Germany, Tokyo, Singapore and São Paulo.