Adding centralized secrets service to Azure Service Fabric cluster
Image from Pexels

Adding centralized secrets service to Azure Service Fabric cluster

Adding secrets store and using store secrets in Azure Service Fabric

Azure Service Fabric is a great platform to host services for your distributed solution. It allows you to easily deploy and scale your applications and services.

Apart from these out of the box functionalities which it is essentially build for, it also comes with secrets service which allows you to store and retrieve secret values for your application. There is no need to use any additional service like Hashicorp Vault or Azure vault if you are ruining your local cluster. 

Setting up Centralized Secrets Service on a local ServiceFabric cluster

I am going to focus on local cluster that is not provisioned and managed in the cloud but rather on on-prem physical or virtual machines managed by you or your team.

Now assuming you already have your local cluster on 3+ nodes, your config probably looks like something like this.

{
"name": "MyServiceFabricCluster",
"clusterConfigurationVersion": "1.0.0",
"apiVersion": "05-2020",
"nodes": [
{
"nodeName": "vm0",
"iPAddress": "NodeVM1",
"nodeTypeRef": "NodeType0",
"faultDomain": "fd:/dc1/r0",
"upgradeDomain": "UD0"
},
{
"nodeName": "vm1",
"iPAddress": "NodeVM2",
"nodeTypeRef": "NodeType0",
            "faultDomain": "fd:/dc1/r1",
            "upgradeDomain": "UD1"
        },
        {
            "nodeName": "vm2",
            "iPAddress": "NodeVM3",
            "nodeTypeRef": "NodeType0",
            "faultDomain": "fd:/dc1/r2",
            "upgradeDomain": "UD2"
        }
    ],
    "properties": {
		"reliabilityLevel": "Bronze",
		"enableTelemetry": "false",
		"fabricClusterAutoupgradeEnabled": false,
        "diagnosticsStore": 
        {
			"metadata":  "Please replace the diagnostics file share with an actual file share accessible from all cluster machines.",
			"dataDeletionAgeInDays": "7",
			"storeType": "FileShare",
			"IsEncrypted": "false",
			 "enableTelemetry": "false",
			"connectionstring": "c:\\ProgramData\\SF\\DiagnosticsStore"
        },
        "nodeTypes": [
            {
                "name": "NodeType0",
                "clientConnectionEndpointPort": "19000",
                "clusterConnectionEndpointPort": "19001",
                "leaseDriverEndpointPort": "19002",
                "serviceConnectionEndpointPort": "19003",
                "httpGatewayEndpointPort": "19080",
                "reverseProxyEndpointPort": "19081",
                "applicationPorts": {
                    "startPort": "20001",
                    "endPort": "20031"
                },
                "ephemeralPorts": {
                    "startPort": "49152",
                    "endPort": "65535"
                },
                "isPrimary": true
            }
        ],
		"healthPolicy": { 
              "maxPercentUnhealthyApplications": "0", 
              "maxPercentUnhealthyNodes": "0" 
        }, 
        "fabricSettings": [
            {
                "name": "Setup",
                "parameters": [
                    {
                        "name": "FabricDataRoot",
                        "value": "C:\\ProgramData\\SF"
                    },
                    {
                        "name": "FabricLogRoot",
                        "value": "C:\\ProgramData\\SF\\Log"
                    }
                ]
            }
        ]
    }
}
    

Before you update your cluster configuration by adding the Centralized Secrets Service to the configuration you need to have a certificate which will be use to encrypt and decrypt your secrets. Since it is used only for this purpose you can use self signed certificate which is quite easy to to create with openssl.

Since openssl is not native Windows application you can either use cywin on Windows, but much better option would be WSL on Windows. One you spin up your Linux terminal, simply replace the values for -subj argument and execute the following command to get the certificate

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt -subj "/C=IL/ST=Oregon/L=Portland/O=MyCompanyName/CN=MyCertificateCommonName/GN=GivenName/SN=SureName/emailAddress=email@addresss.com"
    

You will need to install this on your all your Windows nodes in your local ServiceFabric cluster, so for this purpose you need a PFX format of your certificate.

openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt
    

Now you are ready to install the certificate to your local cluster. Since the service will be live on all nodes on in the cluster you need to install it on all nodes which you can either do through the UI over RDP connection or simply use the PowerShell script.

$password = convertto-securestring "password" -asplaintext -force
Import-PfxCertificate -FilePath "C:\sf\certificate.pfx" -Password $password -CertStoreLocation Cert:\CurrentUser\My -Exportable
    

Before we update the node with new configuration, we'll need to obtain SHA1 fingerprint of installed certificate. You can either do that from Windows by opening your certificate .crt file or from command line in openssl

openssl x509 -sha1 -in certificate.crt -fingerprint -noout
    
Note

openssl exports fingerprint values with colon ":" characters, so to use it in Service Fabric cluster configuration file, you need to remove those colon characters

When certificate is in place on the cluster nodes, we are ready to update the cluster configuration and initialize the cluster update from the elevated permissions PowerShell console. 

Your new cluster configuration needs to be expanded with secrets service section which points to installed certificate via thumbprint string

{
    "name": "MyServiceFabricCluster",
    "clusterConfigurationVersion": "1.0.0",
    "apiVersion": "05-2020",
    "nodes": [
        {
            "nodeName": "vm0",
            "iPAddress": "NodeVM1",
            "nodeTypeRef": "NodeType0",
            "faultDomain": "fd:/dc1/r0",
            "upgradeDomain": "UD0"
        },
        {
            "nodeName": "vm1",
            "iPAddress": "NodeVM2",
            "nodeTypeRef": "NodeType0",
            "faultDomain": "fd:/dc1/r1",
            "upgradeDomain": "UD1"
        },
        {
            "nodeName": "vm2",
            "iPAddress": "NodeVM3",
            "nodeTypeRef": "NodeType0",
            "faultDomain": "fd:/dc1/r2",
            "upgradeDomain": "UD2"
        }
    ],
    "properties": {
		"reliabilityLevel": "Bronze",
		"enableTelemetry": "false",
		"fabricClusterAutoupgradeEnabled": false,
        "diagnosticsStore": 
        {
			"metadata":  "Please replace the diagnostics file share with an actual file share accessible from all cluster machines.",
			"dataDeletionAgeInDays": "7",
			"storeType": "FileShare",
			"IsEncrypted": "false",
			 "enableTelemetry": "false",
			"connectionstring": "c:\\ProgramData\\SF\\DiagnosticsStore"
        },
        "nodeTypes": [
            {
                "name": "NodeType0",
                "clientConnectionEndpointPort": "19000",
                "clusterConnectionEndpointPort": "19001",
                "leaseDriverEndpointPort": "19002",
                "serviceConnectionEndpointPort": "19003",
                "httpGatewayEndpointPort": "19080",
                "reverseProxyEndpointPort": "19081",
                "applicationPorts": {
                    "startPort": "20001",
                    "endPort": "20031"
                },
                "ephemeralPorts": {
                    "startPort": "49152",
                    "endPort": "65535"
                },
                "isPrimary": true
            }
        ],
		"healthPolicy": { 
              "maxPercentUnhealthyApplications": "0", 
              "maxPercentUnhealthyNodes": "0" 
        }, 
        "fabricSettings": [
            {
                "name": "Setup",
                "parameters": [
                    {
                        "name": "FabricDataRoot",
                        "value": "C:\\ProgramData\\SF"
                    },
                    {
                        "name": "FabricLogRoot",
                        "value": "C:\\ProgramData\\SF\\Log"
                    }
                ]
            },
			{
				"name":  "CentralSecretService",
				"parameters":  [
					{
						"name":  "DeployedState",
						"value":  "enabled"
					},
					{
						"name" : "EncryptionCertificateThumbprint",
						"value": "737d74e9b62417becf46b9a5a891fd19eb5a1d49"
					},
					{
						"name":  "MinReplicaSetSize",
						"value":  "1"
					},
					{
						"name":  "TargetReplicaSetSize",
						"value":  "3"
					}
				]
			}
        ]
    }
}
    

New configuration is ready to be applied via simple PowerShell command

# Connect to cluster
Connect-ServiceFabricCluster -ConnectionEndpoint "NodeVM1:19000"

# Initialize cluster update
Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath .\ClusterConfigWithSecretsService.json
    

Once cluster upgrade is finished, you should be able to see CentralSecretService in you Service Fabric cluster dashboard

Sf Css Expand 

Using Centralized Secrets Service to store and retrieve secrets in .NET application

We have secrets service in place in our SF cluster and we should take advantage of it in our application. Configuration values which are environment specific and which are not to be shared with public such as database connection strings or credentials should not be stored anywhere as a plain text values.

I will use example of .NET5 ASP.NET WebAPI application which will access encrypted configuration value. Since we can override configuration values with environment variables in Asp.NET and Asp.NET Core, I'll use this to alter service fabric application ApplicationManifest.xml and ServiceManifest.xml file during the release with PowerShell script.

You can use the same logic regardless of what kind of release pipeline you use. I wrote this one for Azure DevOps release pipeline, but with small changes it can be used easily with Jenkins or TeamCity releases

We'll first update ApplicationManifest.xml file which sits in ServiceFabric application project.


$doc = new-object System.Xml.XmlDocument  

$file = "$(System.DefaultWorkingDirectory)/_BuildPipelineName/drop/applicationpackage/ApplicationManifest.xml"
$thumbprint = "$(CertificateThumbprint)"
$doc.load($file)  

$nsmanager = new-object System.Xml.XmlNamespaceManager($doc.NameTable);  

$applicationManifestNode = $doc.ApplicationManifest 

$certificatesNode = $applicationManifestNode.Certificates

if($certificatesNode -eq $null)
{
    $certificatesNode = $doc.CreateElement("Certificates", $applicationManifestNode.NamespaceURI)  
    $applicationManifestNode.AppendChild($certificatesNode) 
}

$secretCertificateNode = $doc.CreateElement("SecretsCertificate", $applicationManifestNode.NamespaceURI)
$secretCertificateNode.SetAttribute("X509FindValue",$thumbprint)
$secretCertificateNode.SetAttribute("Name","MyCertificateCommonName")
$certificatesNode.AppendChild($secretCertificateNode)

echo $doc.OuterXml

$doc.Save($file)
    

For ServiceManifest.xml, apart from environment variables to be set that will override configuration values, we also set the environment name with ASPNETCORE_ENVIRONMENT key, but you can notice that this one is declared as PlainTex text. This means it is not a secret and we do not need to encrypt it when used in Service Fabric cluster. Value is used as is

$doc = new-object System.Xml.XmlDocument  
$file = "$(System.DefaultWorkingDirectory)/_BuildPipelineName/drop/applicationpackage/MyService.ApiPkg/ServiceManifest.xml"
$environmentVariables =  @(
     [pscustomobject]@{Name='ASPNETCORE_ENVIRONMENT';Type='PlainText'},
     [pscustomobject]@{Name='ConnectionStrings__ApplicationDatabase';Type='Encrypted'}
)

$doc.load($file)  
$nsmanager = new-object System.Xml.XmlNamespaceManager($doc.NameTable)
$nsmanager.AddNamespace("df", "http://schemas.microsoft.com/2011/01/fabric")


$serviceManifestNode = $doc.ServiceManifest
$codePackageNode = $serviceManifestNode.SelectSingleNode("/df:CodePackage[@Name='Code']",$nsmanager)
$environmentVariablesNode = $doc.SelectSingleNode("/df:ServiceManifest/df:CodePackage[@Name='Code']/df:EnvironmentVariables",$nsmanager);

$environmentVariablesNode.RemoveAll();

foreach($environmentVariable in $environmentVariables)
{   

    $environmentVariableNode = $doc.CreateElement("EnvironmentVariable", $serviceManifestNode.NamespaceURI)  
    $environmentVariableNode.SetAttribute("Name",$environmentVariable.Name)
    $environmentVariableNode.SetAttribute("Type",$environmentVariable.Type)
    $environmentVariableNode.SetAttribute("Value",[Environment]::GetEnvironmentVariable($environmentVariable.Name))
    $environmentVariablesNode.AppendChild($environmentVariableNode) 

}

echo $doc.OuterXml

$doc.Save($file)
    

However, environment value under key ConnectionStrings__ApplicationDatabase is set as Encrypted type. This means when acquiring this value from our application, Service Fabric will decrypt it first and then provide it to the application.

Since the value needs to be encrypted in the first place, we need to calculate encrypted value on one of the nodes of our cluster via PowerShell script.

# Connect to cluster
Connect-ServiceFabricCluster -ConnectionEndpoint "NodeVM1:19000"

# Encrypt text value
Invoke-ServiceFabricEncryptText -CertStore -CertThumbprint "737d74e9b62417becf46b9a5a891fd19eb5a1d49" -Text "Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword;" -StoreLocation CurrentUser -StoreName My
    

This value can now be used to set environment variable value and in our case, release pipeline variable. Our code does not require any change and we can scope pipeline variables to specific stage so that we can have different values applied to ApplicationManifest.xml and ServiceManifest.xml depending to which environment we are releasing to.

References

Disclaimer

Purpose of the code contained in snippets or available for download in this article is solely for learning and demo purposes. Author will not be held responsible for any failure or damages caused due to any other usage.


About the author

DEJAN STOJANOVIC

Dejan is a passionate Software Architect/Developer. He is highly experienced in .NET programming platform including ASP.NET MVC and WebApi. He likes working on new technologies and exciting challenging projects

CONNECT WITH DEJAN  Loginlinkedin Logintwitter Logingoogleplus Logingoogleplus

.NET

read more

JavaScript

read more

SQL/T-SQL

read more

Umbraco CMS

read more

Comments for this article