Tag Archives: Azure

[azure application service] the application deployed under the app service reports an error out of memory

Problem description

After the app is deployed to the app service, an out of memory error is encountered.

 

Error message: GetData   Error:, Exception of type ‘System.OutOfMemoryException’ was thrown.,

at System.IO.MemoryStream.set_ Capacity(Int32 value)

at System.IO.MemoryStream.EnsureCapacity(Int32 value)

at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)

at OfficeOpenXml.Packaging.ZipPackage..ctor(Stream stream)

at OfficeOpenXml.ExcelPackage.ConstructNewFile(String password)

at OfficeOpenXml.ExcelPackage..ctor(FileInfo newFile)

at MyLife.Utility.ExcelHelper.LoadDataTableFromFile(String fileName)

at MyLife.Serivces.GetData()

 

2021-09-12 05:19:57.0362 | Error | HCPService-StartPreprocessing-Exception of type ‘System.OutOfMemoryException’ was thrown.-

2021-09-12 05:19:57.0756 | Error | An unhandled exception has occurred while executing the request. System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown.

Problem solving

In the case of oom error at the beginning of deployment, it means that the memroy of the current pricing layer is not enough to meet the operation of the application. However, the memory of different pricing layers of APP service is different. The lowest free layer has 1g ram and the highest can reach 14g.

  （Source ：   https://www.azure.cn/pricing/details/app-service/ ）  

 

If the memory limitation of the pricing layer is excluded, another important factor is the platform setting of APP service.

App service supports 32-bit and 64 bit platform systems. Therefore, if the default setting is 32-bit operating system, it only supports 4G memory at most. In order to make the application use the maximum memory and not waste the ram resources of APP service, when facing the problem of oom, modify the 64 bit platform bit of APP service at the first time. If oom still occurs, you can consider upgrading the pricing layer of APP service.

 

[END]

 

[Solved] Service Fabric VMExtensionProvisioningError Error: Multiple VM extensions failed to be provisioned on the VM

Problem description

Deployment   Vmextensionprovisioningerror was encountered while using azure service fabric. The full text is as follows:

Deployment Name: 385A084F35B2BC55-xxxx
Operation Id: 096CED
Operation: Create
Correlation Id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Status: Failed
Error:
Code: VMExtensionProvisioningError
Message: Multiple VM extensions failed to be provisioned on the VM. 
Please see the VM extension instance view for other failures. The first extension failed due to the error: 
VM Extension 'RunPowerShell_Frontdoor' is marked as failed since it depends upon the VM Extension 'ServiceFabricNodeVmExt_vmNodeType1Name' which has failed.
Help Link: https://aka.ms/ev2/errors/troubleshooting

Problem solving

Step 1: collect logs

In the node that requires remote connection (RDP) to SF, in the directory   C: Collect VM extension installation logs in \ windowsazure \ logs \ plugins

Step 2: analyze logs

stay   In the microsoft.azure.servicefabric.servicefabricnode plugin log, it is found that the error occurred due to the failure to obtain the certificate.

[09/09/2021 12:32:01.86] Executing: C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\ServiceFabricExtensionHandler.exe enable
[09/09/2021 12:32:03.62] Execution Complete.
######
Execution Output:
1> 2021-09-09T12:32:01.9973303Z: Information: Starting...
1> 2021-09-09T12:32:01.9973303Z: Information:   MachineName:       Frontdoor000000
1> 2021-09-09T12:32:01.9973303Z: Information:   CurrentDirectory:  C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11
1> 2021-09-09T12:32:01.9973303Z: Information:   UserName:          SYSTEM
1> 2021-09-09T12:32:01.9973303Z: Information: Created mutex; this is the only instance of this process running
1> 2021-09-09T12:32:02.3723445Z: Information: HandlerManifest Version: 1
1> 2021-09-09T12:32:02.4348520Z: Information: [HandlerEnvironment]
1> 2021-09-09T12:32:02.4348520Z: Information:   LogFolder:     C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11
1> 2021-09-09T12:32:02.4348520Z: Information:   ConfigFolder:  C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\RuntimeSettings
1> 2021-09-09T12:32:02.4348520Z: Information:   StatusFolder:  C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\Status
1> 2021-09-09T12:32:02.4348520Z: Information:   HeartbeatFile: C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\Status\HeartBeat.Json
1> 2021-09-09T12:32:02.4348520Z: Information:   DeploymentId:  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
1> 2021-09-09T12:32:02.4348520Z: Information:   RoleName:      _Frontdoor_0
1> 2021-09-09T12:32:02.4348520Z: Information:   Instance:      _Frontdoor_0
1> 2021-09-09T12:32:02.5442327Z: Information: Opening config file: C:\Packages\Plugins\Microsoft.Azure.ServiceFabric.ServiceFabricNode\1.1.0.11\RuntimeSettings\1.settings
1> 2021-09-09T12:32:02.5598642Z: Information: Config specified client cert FindBySubjectName 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' (store name = My)
1> 2021-09-09T12:32:02.5754780Z: Error: ERROR: Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadClientCertificate(String findValue, String certStoreName, X509FindType findBy)    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadCertificateWrapper(CertificateSettings certificateSetting)
1> 2021-09-09T12:32:02.5754780Z: Warning: Heartbeat: Not Ready: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'
1> 2021-09-09T12:32:02.5754780Z: Error: Heartbeat file updated, with error msg
1> 2021-09-09T12:32:02.5911044Z: Error: ERROR: Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadClientCertificate(String findValue, String certStoreName, X509FindType findBy)    at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadCertificateWrapper(CertificateSettings certificateSetting)    at Microsoft.Azure.ServiceFabric.Extension.Core.Common.PublicSettingsExtensions.AllowAccessToCerts(PublicSettings publicSettings, ICertificateUtility certificateUtility)    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.ValidateDeployment(HandlerSettings settings)    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallService()    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallServiceWithRetry()    at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.Enable()    at Microsoft.Azure.ServiceFabric.Extension.Handler.Program.Main(String[] args) in X:\bt\1162492\repo\src\HandlerExe\Program.cs:line 52


Execution Error:
Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with 'FindBySubjectName' 'xxxxx.cluster.prod.ngpproxy.microsoftonline.cn' in store 'My'
   at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadClientCertificate(String findValue, String certStoreName, X509FindType findBy)
   at Microsoft.Azure.ServiceFabric.Extension.Core.CertificateUtility.LoadCertificateWrapper(CertificateSettings certificateSetting)
   at Microsoft.Azure.ServiceFabric.Extension.Core.Common.PublicSettingsExtensions.AllowAccessToCerts(PublicSettings publicSettings, ICertificateUtility certificateUtility)
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.ValidateDeployment(HandlerSettings settings)
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallService()
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.InstallServiceWithRetry()
   at Microsoft.Azure.ServiceFabric.Extension.Core.VMExtensionHandler.Enable()
   at Microsoft.Azure.ServiceFabric.Extension.Handler.Program.Main(String[] args) in X:\bt\1162492\repo\src\HandlerExe\Program.cs:line 52

The key message is   Microsoft.Azure.ServiceFabric.Extension.Core.AgentException: Cannot find certificate with ‘FindBySubjectName’ ‘xxxxx.cluster.prod.ngpproxy.microsoftonline.cn’ in store ‘My’ 。

Step 3: continue to find the problem that the certificate cannot be found

In the log of obtaining the certificate, it is found that SF is the certificate obtained from kV (key vault). When obtaining the auth challenge in step 5, it took more than 5 minutes from 18:59:02 to 19:04:45.

2021-09-09 18:59:02: < info> [CertificateManager] Beginning refresh for: https://xxxxx-chinanorth-kv.vault.azure.cn/secrets/cluster

2021-09-09 18:59:02: < info> [WindowsCertificateStore] attempting to open store ‘LocalMachine\MY’

2021-09-09 18:59:02: < debug> [WindowsCertificateStore] opening the ‘LocalMachine’ store..

2021-09-09 18:59:02: < debug> [WindowsCertificateStore] store opened successfully.

2021-09-09 18:59:02: < info> [KeyVaultClient] Getting new auth challenge

2021-09-09 19:04:45: < error> [CertificateManager] Refreshing ‘ https://xxxxx-chinanorth-kv.vault.azure.cn/secrets/cluster ‘ failed with KeyVault exception: KeyVaultHttpClient.GetChallenge. – http_ exception: what = ‘WinHttpSendRequest: 12030: The connection with the server was terminated abnormall’ code = ‘windows:12030’

2021-09-09 19:04:45: < info> [CertificateManager] Beginning refresh for:

It can be inferred that when SF is deployed due to network delay, authentication is requested from key vault, and deployment fails due to request timeout.

Redeployment of SF was successful after multiple retries.

[azure environment] notification hub failed to create policy: 500 internal server error occurred

Problem description

There is a problem with the notification hub and the policy (rule) cannot be created. Error message obtained:

Error: Error making Read request on Authorization Rule “notification-hub-auth-rule” (Notification Hub “nh-verify-notify”/Namespace “nh-verify-notify-namespace”/Resource Group “nh-state”) : notificationhubs.Client#GetAuthorizationRule: Failure responding to request: StatusCode=500 — Original Error: autorest/azure: error response cannot be parsed: “\”Internal Server Error\”” error: json: cannot unmarshal string into Go value of type azure.RequestError

When creating a policy through the azure notification hub portal, you encounter the same problem:

 

 

Question answer

After locating with the help of azure support, it is proved that this problem is a bug after the upgrade of notification hub service. Therefore, when a service on azure encounters a problem, after preliminarily eliminating the use error, the fastest solution is to find azure support to solve the problem:

https://support.azure.cn/zh-cn/support/support-azure