SSIS with AlwaysOn

In SQL Server 2012, AlwaysOn Availability Groups maximizes the availability of a set of user databases for an enterprise. In addition, SSIS introduces new capabilities which allow customers with the ability to easily deploy to a centralized SSIS Catalog (i.e. SSISDB user database).

In this article, we explore how you can make use of SSIS and AlwaysOn Availability Groups. Using these capabilities, it allows you to provide high-availability of SSISDB and its contents (projects, packages, execution logs, etc).

SSISDB in Availability Groups

To use SSIS with AlwaysOn, you need to add the user database SSISDB into Availability Groups. Please note there are prerequisites on Windows System and SQL Server instances to use availability groups. With these prerequisites met, below are steps to setup SSIS with AlwaysOn:

1. Connect to every node and create the SSISDB catalog. We need to create the catalog even on secondary nodes, because there’re other server-level configurations (cleanup jobs, keys, accounts etc) besides the SSISDB database. Then delete the SSISDB databases on secondary nodes.
2. Create an availability group, specifying SSISDB as the user database; specify secondary replicas.

Failover

With SSISDB being setup as part of an availability group, you can now leverage the failover capabilities provided by AlwaysOn Availability Groups. When a failover occurs, one of your secondary nodes now becomes the new primary node.

Currently you may get an error 15581 when you try to run a package on the new primary node:

“Please create a master key in the database or open the master key in the session before performing this operation”.

This is due to that the SSISDB database master key needs to be re-encrypted by the server master key after a failover. This error could occur when operations try to access encrypted data, such as in running a package or deploying a project. To solve it, you can manually re-encrypt the database master key by running this T-SQL snippet:

USE SSISDB　　　　　　 
 Open master key decryption by password = 'x' -- Password used when creating SSISDB'
 Alter Master Key Add encryption by Service Master Key
 

Now everything should work fine.

Auto-Detecting Failover

How to automatically run above script when a failover occurs? Today, there are no straightforward capabilities for determining that a failover has occurred. You can consider the following workaround before creating the availability group:

1. On every node, create a tiny table containing only 1 row in a database other than SSISDB (say, X):

USE X;
 CREATE TABLE [dbo].[replica_role](
 [replica_role] [tinyint] NULL
 ) ON [PRIMARY]
 INSERT INTO [dbo].[replica_role] ([replica_role]) (SELECT [role] FROM sys.dm_hadr_availability_replica_states WHERE is_local = 1);
 

      2.    On every node, create an agent job (say, “ssisdb_failover”) with a recurring step that runs below T-SQL script every certain amount of time (say, every 15 minutes):

USE X;
 
 DECLARE @last_role TINYINT;
 SET @last_role = (SELECT TOP 1 [replica_role] FROM [dbo].[replica_role]);
 
 DECLARE @current_role TINYINT;
 SET @current_role = (SELECT ROLE FROM sys.dm_hadr_availability_replica_states WHERE is_local = 1);
 
 IF (@last_role = 2 AND @current_role = 1) --Last time it was secondary, currently it is primary: need re-encrypt the database master key
 BEGIN
 USE SSISDB;
 OPEN MASTER KEY DECRYPTION BY PASSWORD = 'x' –Should be encrypted! See below
 ALTER MASTER KEY ADD ENCRYPTION BY SERVICE MASTER KEY
 END
 
 USE X;
 UPDATE dbo.[replica_role] SET [replica_role] = @current_role;
 

Encrypting the Password for the Database Master Key

In the above script, the password for the database master key is in plain text. As a security best practice, you can store it in a table using a simple symmetric key encryption, and retrieve it later. To encrypt and store the password, append below lines to the script in above Step 1:

USE X;
 -- Add a column in which to store the encrypted data.
 ALTER TABLE dbo.replica_role
 ADD ssisdb_pwd VARBINARY(128); 
 GO
 
 --If there is no master key for X, create one now. 
 IF NOT EXISTS 
 (SELECT * FROM sys.symmetric_keys WHERE symmetric_key_id = 101)
 CREATE MASTER KEY ENCRYPTION BY 
 PASSWORD = '23987hxJKL95QYV4369#ghf0%lekjg5k3fd117r$$#1946kcj$n44ncjhdlj' --Your master key password
 GO
 
 CREATE CERTIFICATE ssisdb_cert
 WITH SUBJECT = 'SSISDB Password';
 GO
 
 CREATE SYMMETRIC KEY ssisdb_key
 WITH ALGORITHM = AES_256
 ENCRYPTION BY CERTIFICATE ssisdb_cert;
 GO
 
 -- Open the symmetric key with which to encrypt the data.
 OPEN SYMMETRIC KEY ssisdb_key
 DECRYPTION BY CERTIFICATE ssisdb_cert;
 
 -- Encrypt the value in the column using the symmetric key.
 UPDATE dbo.replica_role
 SET ssisdb_pwd = EncryptByKey(Key_GUID('ssisdb_key'), N'x');
 GO
 

And to retrieve the password, replace the line “OPEN MASTER KEY DECRYPTION BY PASSWORD = 'x'” in the agent job script with below lines:

USE X;
 OPEN SYMMETRIC KEY ssisdb_key DECRYPTION BY CERTIFICATE ssisdb_cert; 
 DECLARE @pwd NVARCHAR(MAX);
 SET @pwd = (SELECT TOP 1 CONVERT(NVARCHAR,DecryptByKey(ssisdb_pwd)) FROM dbo.replica_role);
 DECLARE @sqlString NVARCHAR(1024);
 SET @sqlString = 'OPEN MASTER KEY DECRYPTION BY PASSWORD = ''' + @pwd +'''';
 USE SSISDB;
 EXECUTE sp_executesql @sqlString;
 


 
Note, this workaround can also be applied to auto-detect the failover of other databases.

Other Considerations

When an unplanned failover occurs, it is important that the status of SSIS package executions reflect the most up-to-date state. You can restart the instance so that a default SSIS cleanup job will be executed to fix the status of any packages/ deployments that were running at the time the instance went down. The status will be changed to Terminated; temporary data created during project deployment will be removed.

Can we automatically restart packages after failover? This article provides a good start.