Resource Not Found (dependsOn is not working)

I’ve been getting this question quite a bit lately so I thought a good reason to get something written down (and it forced me to realize I never finished the previous topic).

You have a resource in an Azure Resource Manager template that references another resource using reference() or listKeys(). That reference() call is behind a dependsOn or a condition property, but the deployment tries to reference the resource too early and the deployment fails. This makes it look like the dependsOn is ignored or in the case of a condition, the resource may not even be deployed but the reference() or listKeys() is still called. Here’s how to fix it…

First some background – these functions reference() or anything that starts with list*() are what we call “run-time” functions – meaning they are evaluated at “run-time” or when the deployment has started. All the other functions and language expressions in ARM are “design-time” or “compile-time” functions. They are evaluated before deployment begins.

Once deployment starts, all run-time functions are scheduled by the deployment engine and they can be done in parallel to other operations. It’s this parallel execution, that keeps things fast and can occasionally cause the problem. Sometimes the scheduler doesn’t honor the dependency and in the case of a condition, ARM can evaluate the resources (partially) even if the resource will not be deployed.

To be clear, this is a really old design pattern and the capabilities of the language have grown up around it and exposed some scenarios as deployments get more sophisticated. We’re working on fixing it but it’s harder than it sounds and you can work around it, in some cases pretty easily if you know how it works. Since there are multiple things we need to work through here you may not see the exact symptoms as they start to go away (as we do start to fix things). We’re going through this slowly since this deployment engine handles millions of operations every day. But since it’s a question I keep getting here’s how you can “fix” it – try the following (in this order).

  1. Remove the apiVersion from the reference() call. This *only* works if the resource being referenced is deployed unconditionally in the same template. If the apiVersion is omitted, ARM assumes the resource is defined in the template and schedules the call after the resource is deployed. If the resource is conditionally deployed you can’t do this since the apiVersion is needed for the GET (i.e. the reference()) and the apiVersion on the resource is actually not available (remember I mentioned this was harder to fix than it sounds). Also, this doesn’t work for any list*() function as the apiVersion is always required in a list*() function.
  2. Wrap the call in an if() statement. In the case of a conditional resource, wrap the run-time function call in an if() statement with the same condition as the resource itself. ARM won’t evaluate the “false” side of the statement so the call is never made – like you would expect with the conditional resource itself (hopefully you can start to see why this is hard).
  3. Use a nested deployment. If neither of the above work, you likely have a scenario where the resource is conditionally deployed as part of a new or existing pattern. I.e. the template needs a storage account but it may or may not be deployed in the same template. In this case, you need the apiVersion parameter, which means ARM will schedule the call when it can and sometimes that happens too early. The only way around this last scenario is to nest the deployment that needs the reference() or consumes the resource. In that case, you can do one of two things, which I’ll show below. Nesting the deployment schedules another deployment in ARM and if dependencies are set *and* run-time functions are used as input to the deployment, ARM will defer the call. Knowing this you can use it for other “advanced” scenarios too, like changing deployment options based on run-time state (more on this below too).

Remove API Version

Remember this only works with reference(), not list*(), and when the resource is unconditionally deployed in the same template, but it’s this simple:

        "diagnosticsProfile": {
          "bootDiagnostics": {
            "enabled": true,
            "storageUri": "[reference(variables('storageAccountName')).primaryEndpoints.blob]"
          }
        }

Also, if the name of the resource is unambiguous in the template (i.e. unique) then the full resourceId is not needed and you can use use the resource name.

Use an if() wrapper

If you have a conditional resource, just wrap the run-time function in an if() statement with the same condition, it will only be evaluated when the resource is actually deployed. See line 2 and line 13.

    {
      "condition": "[parameters('deployThisResource')]",
      "type": "Microsoft.Web/sites",
      "apiVersion": "2019-08-01",
      "name": "[variables('webSiteName')]",
      "location": "[parameters('location')]",
      "properties": {
        "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
        "siteConfig": {
          "appSettings": [
            {
              "name": "CosmosDb:Key",
              "value": "[if(parameters('deployThisResource'), listKeys(resourceId('Microsoft.DocumentDb/databaseAccounts', variables('cosmosAccountName')), '2020-04-01').primaryMasterKey, 'does not matter')]"
            }
          ]
        }
      },

Use a Nested Deployment

Ok, this is the most inelegant of the three but if your scenario isn’t handled by the other two above, this will always work. The other technique you can use this approach for is a scenario where you want to know something about a resource (or a dependency) before deploying it. Remember the nested deployment scheduling is “strict” so things will always be done in the order you expect. So whether it’s just a simple case of sequencing or you need to retrieve run-time state from a resource before you know what you want to deploy, this technique can be used.

The key here is that you are using a run-time function as a parameter to a nested deployment. This could be referencing a deployment output from another nested deployment or any other run-time function as a parameter value that’s passed in to the nested deployment. The nested deployment could be a linked template or inline, doesn’t matter, they will behave the same way.

Here is one example – I linked to the line of code, but if (or when) it changes look at the MongoDBUri parameter value. That param value is calling the list*() function directly which will defer the call. Another way you could do the same thing is just pass in the resourceId() and put the listKeys call in the template which is a slightly better practice for debugging.

Here’s is a more advanced example. In this example, I want to set up diagnostic logging on storage endpoints, but not all storageAccounts expose all endpoints. To simplify deployment, I can just prompt for the storageAccount name and determine the endpoints it supports at run-time – rather than have the user of the template supply the endpoints (and potentially miss one and it wouldn’t be logged).

Key points here:

  • This nested deployment is inline, not linked, so no staging required. expressionEvaluationOptions are set to inner scope (line 9), so evaluation is deferred until needed.
  • One of the parameters to the template is an object containing all endpoints available on the storageAccount (line 13), so I don’t need to know ahead of time which are supported on the given storageAccount.
  • Variables then make it easy to set the condition property on each diags resource to determine which endpoints need logging (line 34, etc).
    "resources": [
        {
            "apiVersion": "2019-10-01",
            "name": "nested",
            "type": "Microsoft.Resources/deployments",
            "properties": {
                "mode": "Incremental",
                "expressionEvaluationOptions": {
                    "scope": "inner"  // this allows putting any template inline and evaluation is deferred
                },
                "parameters": {
                    "endpoints": {
                        "value": "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2019-06-01', 'Full').properties.primaryEndpoints]"
                    }
                   // snip
                },
                "template": {
                    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
                    "contentVersion": "1.0.0.0",
                    "parameters": {
                        "endpoints": {
                            "type": "object"
                        }
                    //snip
                    },
                    "variables": {
                        "hasblob": "[contains(parameters('endpoints'),'blob')]",
                        "hastable": "[contains(parameters('endpoints'),'table')]",
                        "hasfile": "[contains(parameters('endpoints'),'file')]",
                        "hasqueue": "[contains(parameters('endpoints'),'queue')]"
                    },
                    "resources": [
                        {
                            "condition": "[variables('hasblob')]",
                            "type": "Microsoft.Storage/storageAccounts/blobServices/providers/diagnosticsettings",
                            "apiVersion": "2017-05-01-preview",
                            "name": "[concat(parameters('storageAccountName'),'/default/Microsoft.Insights/', parameters('settingName'))]",
                            "properties": {
                                "workspaceId": "[parameters('workspaceId')]",
                                "storageAccountId": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageSinkName'))]",
                                "logs": [
                                    {
                                        "category": "StorageRead",
                                        "enabled": true
                                    }
                                ],
                                "metrics": [
                                    {
                                        "category": "Transaction",
                                        "enabled": true
                                    }
                                ]
                            }
                        },
                        {
                            "condition": "[variables('hastable')]"
                            // snip
                        },
                        {
                            "condition": "[variables('hasfile')]"
                            // snip
                        },
                        {
                            "condition": "[variables('hasqueue')]"
                            // snip
                        }
                    ]
                }
            }
        }
    ]

I think that captures it… again apologies for this being necessary – we are working on it but in the meantime…

As always, lmk if I missed anything or there’s something else you’d like to see…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: