You may see the message Code: 9999, Error: Execution error in the pm_error.log file during an SSH key rotation or change, with no obvious hints as to the cause. One possible reason for this issue is related to the SSH key format. Starting from OpenSSH 7.8, the private key is not output in OpenSSL’s PEM format by default, which is one the format accepted by CyberArk as an SSH key.
Continue reading “CyberArk Account Password Change Failed with Code: 9999, Error: Execution error”Category: Troubleshooting
Workaround to connect the server with RDP Licenses not available Error
You might sometimes see the error like this when trying to connect to Remote Desktop Session Host server
The remote session was disconnected because there are no Remote Desktop License Servers available to provide a license.
Continue reading “Workaround to connect the server with RDP Licenses not available Error”Resolving winget not recognized error when running with the System Account
Although winget exists on your system, but when you try to run the winget with system account (or using the scheduled task with the system account) and you see this error.
winget : The term ‘winget’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
Continue reading “Resolving winget not recognized error when running with the System Account”Esxi Kickstart file in the network location problem during Scripted installation with CDROM
If the ks.cfg file is skipped although it’s placed in a correct network location for the semi-automated installation (booted from CDROM), you may need to check these steps in case you missed them.
Continue reading “Esxi Kickstart file in the network location problem during Scripted installation with CDROM”The WS-Management service cannot process the request. Cannot find the Microsoft.PowerShell session configuration in the WSMan: drive on the…
After updating to Ps 4.0, I found some windows hosts are encountering the following errors when I run Invoke-Command to check the powershell version. See Fig-1.
Connecting to remote server X.X.X.X failed with the following error message : The WS-Management service cannot process the request. Cannot find the Microsoft.PowerShell session configuration in the WSMan: drive on the X.X.X.X computer.
Continue reading “The WS-Management service cannot process the request. Cannot find the Microsoft.PowerShell session configuration in the WSMan: drive on the…”
High Memory Usage with w3wp.exe in Exchange Server
w3wp.exe process is an IIS web application process to handle the client request for the application pool. Exchange server services heavily utilized w3wp process not only to handle users request from external but themselves make web service requests among Exchange server members using virtual directories (Owa, OAB & Powershell etc) and respective App pools. Unless you have not configured periodic recycling for Application Pool, you may need to do manual recycle to avoid memory leaks. Microsoft Technet states that: Continue reading “High Memory Usage with w3wp.exe in Exchange Server”
Recover Crashed Exchange 2013 Mailbox Server in DAG
Recovering a crashed mailbox server is a straight-forward process if they are in DAG. You can do it by setup.exe /m:RecoverServer. However, there are certain steps to do for smooth recovery process. The following steps will do to recover the crashed mailbox servers in DAG. I will explain the each steps in more details.
-
Reset the crashed computer accounts in AD.
-
Install new server OS to replace the old crashed servers. Install windows features, pre-requisites and updates.
-
Remove the database passive copies on crashed servers. If the servers are accessible you can manually delete DB file and logs file residues from crashed servers.
-
Remove the crash servers from DAG. This can be done by EMC or EMS.
-
Evict(remove) the crash servers from failover cluster manager.
-
Start the recovery process by running setup file in command prompt with necessary switches. More details later in this section.
Here are detail steps I did for recovery:
1. Reset the crashed computer accounts in AD.
You can do it in ADUC console. Right-click the crashed computer account and “Reset Account”.
2. Install new server OS to replace the old crashed servers.
Install new servers with the same spec as old ones. You also need to install some windows features ,Microsoft filter packs and Unified Communication Runtime.
a) On new machine, open powershell with “Run as Administrator.
b) Install the necessary windows features in Powershell:
Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation c) Download and install other prerequisities from here:
Note: Use the same computer name as old ones and join to domain.
3. Remove the database passive copies on crashed servers.
You can do it on EMS on good servers(since bad servers are not accessible). Open EMS and type:
[Ps]C:> Get-MailboxDatabaseCopyStatus *<name of your crashed server>
Make sure all the listed selected databases on your console output are all in Service Down state. Now, you can remove the failed databases by the following command. This should give you some warnings and don’t worry, just proceed it.
[Ps]C:> Get-MailboxDatabaseCopyStatus *<name of your crashed server> | Remove-MailboxDatabaseCopy
4. Remove the crash servers from DAG.
You can remove crashed servers from DAG by EMC console.
Go to EMC >> servers >> database availability group. Select the DAG Group and click the “Manage DAG Membership” icon. Remove the crashed servers from there.
<or>
You can remove crashed servers in EMS shell.
Remove-DatabaseAvailabilityGroupServer -Identity <your DAG Name> -MailboxServer <Your failed server name>
5. Evict the crash servers from failover cluster manager
Removing the failed servers from DAG does not remove them from failover cluster itself. So, we have to manually remove it. Before this, you can check which nodes in cluster are currently down state by the elevated command prompt.
C:cluster.exe node
To evict the node from cluster:
Go to Failover Cluster Manager >> [your cluster name] >> Nodes >> select your failed server >> Right-click and choose “More action” >> Evict
6. Start the recovery process
Go the directory where setup files are located and run:
setup.exe /m:RecoverServer /IAcceptExchangeServerLicenseTerms
setup.exe /m:RecoverServer /IAcceptExchangeServerLicenseTerms
In most cases, when you use the original exchange installation CD for recovery, you might encounter errors that prompts you to use the later cumulative updated exchange setup files than the ones you have setup. If so, you can find the latest released Exchange CUs here, get the latest CU, extract it to folder and run the setup files again with the switches shown above.
You can also check your current exchange server version with build numbers on good servers in EMS shell by:
[PS] C:\>Get-ExchangeServer | Format-List Name, Edition, AdminDisplayVersion
As the recovery process is fetching info from AD objects and reinstalling the exchange server, you can see the progress in the console.
You can also check your current exchange server version with build numbers on good servers in EMS shell by:
[PS] C:\>Get-ExchangeServer | Format-List Name, Edition, AdminDisplayVersion
As the recovery process is fetching info from AD objects and reinstalling the exchange server, you can see the progress in the console.
7) Add to DAG and Re-seeding DB copies
After the servers are recovered, you need to reboot the recovered servers for proper functioning. Then,
a) add the servers back to DAG group.
b) reseed the DB passive copies.
This is quite a simple process and I won’t go details with these.
Congratulation ! Your recovery process is now Successful.
Note: For me, I had some minor issues when reseeding DB with the following errors. So, I had to take additional steps to fix.
ERROR:
The seeding operation failed. Error: An error occurred while performing the seed operation. Error: Unable to delete logs at ‘D:\M datalogs’. The database has been seeded successfully. If any obsolete log files exist, manually delete them to prevent database divergence. Error: System.IO.IOException: The file or directory is corrupted and unreadable. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileInfo.Delete() at Microsoft.Exchange.Cluster.Replay.DatabaseSeederInstance.DeleteLogFiles(DirectoryInfo di, String logfilePrefix, String logfileSuffix, Int32& logNum) at Microsoft.Exchange.Cluster.Replay.DatabaseSeederInstance.
So, I assume there is some disk corruption in my harddisk. Fortunately, that server has no active database copies, So I manually chkdsk the volume without /r switch. Since some disk errors are found, I use the chkdsk /r /f instead.
C:chkdsk d: /r /f
Then, I reseed the DB again, and another errors come up.
Error:
The seeding operation failed. Error: An error occurred while performing the seed operation. Error: Failed to notify source server ‘[my source server name]’ about the local truncation point. Hresult: 0xc8000713. Error: Unable to find the file.
According to this article , it said that this is the issue with the old logs folder which was not in the sync. And, I did the following procedures:
1) Dismount the Active Copy of the database from EMC console.
2) Find the Database file path and Log file folder path in EMS shell. Let us assume here, edb file path is D:\Databasemydatabase.edb and Log folder path is D:\DatabaseLogs.
[PS]C:>Get-MailboxDatabase mdb04 | fl *path*
3) Login to the source server hosting that database, here ‘[my source server name]’ and you need to run eseutil.exe in elevated command prompt to verify that DB is in clean shutdown state. We have got the DB path & log folder path in step 2.
C:\eseutil /mh D:Databasemydatabase.edb
(If the DB state is clean shutdown, you can continue the next step. If the state is dirty shutdown, you need to go for the recovery process using log files. And this article will help you.)
4) If the DB is clean shutdown state, you can delete all log files in folder path we obtained from step-2. If you are unsure, you can rename the Log folder and delete them later. You can also delete via the following command if there are thousands of log files.
D:\DatabaseLogsrmdir . /s /q
5) Mount the database, this will create new log files.
6) Reseed the database copies.
Change grayed out Windows Service Startup Option
You might sometimes encounter grayed-out services, particularly in scenarios like antivirus programs where certain services are intentionally safeguarded against tampering for security purposes, can pose challenges in managing your system effectively. However, there are strategies you can employ to navigate this hurdle and regain control over these services.
Option 1 – Startup Config
1) type “msconfig” in Run box
2) in the service tab, uncheck the service
3) reboot the computer
Option 2 – Registry Modification
The second method involves accessing the Windows Registry, the central repository of system settings, and making targeted modifications to alter the startup type of the grayed-out services.
1) Go to HKLMSYSTEMCurrentControlSetServices
2) Double-Click the Start SubKey
3) Change the DWORD value to 0 to 4 according to your startup option. 2 for Automatic & 4 for Disabled.
Below are Start values and description according to the technet article.
| Value | Description |
|---|---|
| 0 | Boot (loaded by kernel loader). Components of the driver stack for the boot (startup) volume must be loaded by the kernel loader. |
| 1 | System (loaded by I/O subsystem). Specifies that the driver is loaded at kernel initialization. |
| 2 | Automatic (loaded by Service Control Manager). Specifies that the service is loaded or started automatically. |
| 3 | Manual. Specifies that the service does not start until the user starts it manually, such as by using Device Manager. |
| 4 | Disabled. Specifies that the service should not be started. |