Migrating Oracle Applications into AWS
As an organization, Delphix takes pride in building products that enable our customers to drive innovation and growth. Our Agile Data Platform enables businesses to adopt Agile development methodologies by removing the constraints around data.
The Compliance Engine, released a few weeks ago, is the next step in that progression. Deploying in Amazon Web Services (AWS) and vCloud, the Compliance Engine removes infrastructure constraints that stymie growth. Using this platform, organizations can safely and securely expand their environments into the cloud. Delphix technologies like SnapSync, LogSync and Replication will then keep the environments synced up securely and efficiently between on-premise and scale out environments in the cloud.
I experienced a lot of these benefits firsthand while expanding our performance lab into AWS in order to test the Compliance Engine. Once we had the beta port ready for AWS, the first task was to determine the best deployment strategy in AWS. Optimizing storage is key to Delphix's performance, so I started by evaluating the different block level storage options available in AWS. I discussed those results in my previous blog post about Elastic Block Storage.
The next phase was to evaluate the performance of key workflows in Delphix under AWS. We have historical data on the performance of these workflows. In order to do a true apples-to-apples comparison of performance of the new release, I needed to migrate the data used in my lab onto AWS and run the same experiments. That task alone would have been very difficult and tedious without Delphix.
Application Development in the Cloud
After speaking to a few of our existing and potential customers, I heard these concerns come up repeatedly:
- How do we migrate non-production dev/test environments to a cheaper, more efficient cloud infrastructure?
- How do we keep environments in the cloud in sync with on-premise environments without paying an exorbitant data transfer cost?
- How do we manage burst capacity during high load periods without incurring sustained maintenance/infrastructure costs?
- How do we ensure data is moved securely to the cloud and protected once it is in the cloud?
The sheer volume of data involved in application development makes traditional solutions to these tasks difficult and unreliable. In my experience Delphix naturally addresses all these concerns, as you will see through this post.
Expanding into the Cloud
There are two options for moving application development into the cloud using Delphix:
- Direct Sync: Use SnapSync to link your Delphix Engine (DE) in AWS to your on-premise DB.
- Distributed Sync: Use Delphix Replication to replicate the on-premise DE to a DE in AWS. The on-premise DE will sync to the production DB using SnapSync.
Both options use our custom transport protocol DSP for data transfer, so both will get the benefits of DSP: inline encryption and/or compression. I recommend compressing the data stream in order to obtain better end-end throughput and to save on data transfer cost. Both SnapSync and Replication use efficient data transfer, in that they send only change blocks and discard zero or un-used blocks. This will also reduce costs by minimizing the data transferred.
Keeping Cloud and On-premise Environments in Sync
I chose to use the distributed sync option to move my performance regression testing environments into AWS due to a few key advantages it presents over Direct Sync. Figure 1 depicts the architecture I am using to expand my lab in HQ into AWS.
We have two labs, in our Menlo Park and Boston offices. In order to facilitate testing by both the teams, I need to bring up environments closer to both locations. I used a DE in my Menlo Park lab as a portal to migrate the environments to the two DEs in AWS, placed in the "East-1a" and "West-1b" regions. The West-Coast AWS 'lab' acts as the primary testing ground for AWS feature development and QA etc.
I typically have 3-5 environments provisioned in the West coast lab. The DE in this lab is in "Active-Active" mode with the DE in HQ - environments are typically refreshed along with the ones in the Menlo park lab. I also have a secondary lab setup in the East-1a region, replicated from the DE in HQ, but in an "Active-Passive" mode.
The DE in East-1a does not serve any environments under normal conditions. We bring up environments from this engine only during release sprints, etc., when I need more environments than what the on-premise Boston lab can sustain. For all of our customers who encouraged us to support AWS, this type of architecture is ideal. It provides an easy and natural path to migrate/expand their current non-prod environments into a private or public cloud.
Delphix Data Cloud
The on-premise DE acts as a multi-functional portal to all of your cloud environments. I have test environments provisioned from three DEs now: one on-premise in HQ and two in the cloud servicing my teams in two geographical regions. Since the DE on-premise is doing the heavy lifting of replicating the data to the other two DEs, it relieves the production DB of any additional load.
Because of this, the Distributed Sync solution will scale across as many regions as desired. All my environments are now in sync and are working of the same dataset from production. We have the ability to test features from different releases by syncing all our virtual databases in AWS and on-premise to the same snapshot. This operation takes a few minutes and will not involve any data movement across geographical regions.
One of the big drivers for expansion into the cloud is burst capacity: the ability to easily scale resources during high-demand periods without paying a sustained cost for the infrastructure. Because of our short release cycles, I do not have the luxury of low load periods for our testing environments. But we do have bursts of regression testing towards the end of every release.
Until now, the solution has been a painful arbitration of the resources based on the priorities of the different bugs etc. With the expansion of the performance labs into AWS, we can scale up the resources during high load periods and tear them down after the release. Figure 2 shows the architecture during the high load periods.
This is an active-passive architecture during low load periods - the DE in AWS is only receiving periodic Replication updates and stays in sync with the on-premise environments. During this period, there aren't any any environments provisioned off of this engine.
Since the replication updates are asynchronous and not IO intensive, I use an inexpensive instance type for this DE - to keep costs down. This DE is now acting as DR backup for my regression testing non-prod environments. For each release, the DE instance type will be bumped up to a high-performance one and the architecture becomes active-active, with environments provisioned off of both on-premise and AWS DEs.
Security and Data Protection
Another big concern for expanding application development into cloud is security and protection for critical and sensitive application data. I read in a recent survey about cloud computing that security is the biggest entry barrier for 1/3rd of the respondents. Delphix Snapsync and Replication technologies support encryption for moving data between on-premise and cloud environments.
When using AWS, Delphix deploys into a Virtual Private Cloud (VPC) and can be configured to be part of a VPN. All of these technologies protect sensitive application data without any overhead. Though we do not have any sensitive data in our performance lab environments, they are critical to our release cycles. Having all of the environments replicated in AWS gives an invaluable layer of disaster recovery protection for our development and test environments.
In this blog post I talked about my experience in testing the Compliance Engine and how that task was simplified using the Compliance Engine. It was remarkable how easy and natural it was to expand my lab into AWS to service teams in multiple geographical regions. I drew the architecture I used for expanding my performance lab into AWS to service the teams in our two offices. Hopefully you will find parallels to this in your environments and can use this architecture as a cookbook for expanding your environments into cheaper cloud infrastructures.