IT Consultant Everyday Notes

Just some problems/solutions storage

SCCM: Side-by-side migration issue with client reassignment

I was busy with an SCCM migration recently. A Customer wanted to get gradual side-by-side migration from an old SCCM 2012 R2 to a shiny SCCM CB.

The issue I faced was related to a Client re-assignment from the old to the new SCCM site.

As recommended I tried Jason Sandy’s script to reinstall the old client and configure the new one for the new site.

The Client was successfully installed, but kept connect to the old site.

I tried to re-register site assignment in WMI as described https://prajwaldesai.com/change-site-code-of-configuration-manager-client/ and restart CCMEXEC service.

In ClientLocation log I saw the new site was assigned, MP found, but after that the site immediately was re-assigned to an old one and the Client tried to connect back to the old site Sad smile

I tried  completely uninstall the client, use push etc.. without success.

Finally I noted

LSRefreshSiteCode: Group Policy Updated the assigned site code <old site code>, which is different than the existing assigned site code <new site code >. Will attempt re-assignment.

I checked GPOs and found a disabled GPO containing SCCM ADMX template with a site assignment.

The matter in fact once applied GPO tattoes its settings in the registry and they remains there even if GPO is not active anymore.

So I opened GPO template (located in ConfigMgr installation folder\Tools\ConfigMgrADMTemplates and founf the registry key in question is “hklm\SOFTWARE\Microsoft\SMS\Mobile Client”. Originally I planned to change only site code value there, but found Henrik’s article where he recommended to remove all values from the key all together.

Probably both approaches can work, so I created a simple cmd script and pushed it from the SCCM

REG delete “hklm\SOFTWARE\Microsoft\SMS\Mobile Client” /v GPRequestedSiteAssignmentCode /f
REG delete “hklm\SOFTWARE\Microsoft\SMS\Mobile Client” /v GPSiteAssignmentRetryDuration(Hour) /f
REG delete “hklm\SOFTWARE\Microsoft\SMS\Mobile Client” /v GPSiteAssignmentRetryInterval(Min) /f
cscript set-site-code.vbs

first three commands are cleaning settings hardcoded by GPO, the forth one force SCCM site code using a VBS script from here

After the script finished I restarted ccmexec and fount the client registered in the new site successfully.

Advertisements

Azure: SQL Server 2017 Always-On Availability group on Windows Server 2016 hosted by Azure

One of my Customers needed a help with installation of SQL 2017 Always-on Cluster on Windows Server 2016 in Azure (IAAS).

Here is a log of my actions. I used available instructions for SQL 2014 on Windows Server 2012 R2, but added a new feature for Server 2016 – Cloud witness. That allowed my Customer to save on a File Server compute.

It was just one time install, so I did not bother to create an ARM template (unfortunately the one available on GitHub for Server 2012R2 and SQL 2014 SP2 does not work with Server 2016 image)

1. My Customer wanted to use its own licenses, so I decided to use images Microsoft made available for this scenario recently recently. Here is a script to check what BYOL images are available for your region

Get-AzureRMVMImageOffer -Location ‘Eastus’ -Publisher ‘MicrosoftSQLServer’ | Select Offer

I used SQL2017-WS2016-BYOL (the image conveniently has a data drive in addition to System and Scratch drives)

2. Bring two VMs from the image

3. Logon to the VMs and join them to the domain (or add an extension for autojoin)

4. Change Logon accounts for SQL  Server Service and SQL Client Service to  domain accounts (necessary for Clustering)

5. Configure SPN for SQl Server Service account

6. Add ports TCP 1433, 59999 (Azure LB Probe), 5022 (Database Mirroring) to Windows firewall exceptions

5. Restart VMs

6. Add Failover Clustering Feature on both

7. Build a one node Cluster on the first node. DO NOT USE VERIFICATION

8. Create Client Access name (if installation account has enough rights it will be automatically added to the same OU where machine accounts for the nodes are otherwise you need to create it manually).

9. Azure cannot bound multiple IPs to NIC, so Change IP address for Client Access to Manual and set it up to a free IP from your subnet. For that Right-click the failed IP Address resource, and then click Properties

10. Select Static IP Address and specify an available address from subnet where the SQL Server is in the Address text box. Then, click OK.

11. In the Cluster Core Resources section, right-click cluster name and click Bring Online. Then, wait until both resources are online. When the cluster name resource comes online, it updates the DC server with a new AD computer account. Use this AD account to run the Availability Group clustered service later.

12. Add the second node to the Cluster (using Failover Cluster Management Console). Do not do Validation

13. In the Confirmation page if you are using Storage Spaces, clear the checkbox labeled Add all eligible storage to the cluster.

image

14. Add a witness using Azure Storage account as described here: https://docs.microsoft.com/en-us/windows-server/failover-clustering/deploy-cloud-witness

15. Start SQL Server Manager, Goto to Properties of SQLServer (MSSQLSERVER) Click the AlwaysOn High Availability tab, then select Enable AlwaysOn Availability Groups

image

16. Apply and restart SQL Server Service

17. Create an empty database on the first SQL

18. Create a backup share and backup db there (full backup)

19. Restore the full and log backups to the second SQL Server with the NORECOVERY option

20. Create the Availability Group

     – right-click AlwaysOn High Availability and click New Availability Group Wizard  

image

– In the Specify Availability Group Name page, type a name for the Availability Group, for example AG1, in Availability group name.

– In the Select Databases page, select your database. The database meets the prerequisites for an Availability Group because you have taken at least one full backup on the intended primary replica.

– Click Add replica

– Type name of the second server in popup window and click Connect

– Now the second server should be visible in the Specify Replica page

– Click Endpoints to see db mirroring. Check the port!

– In the Select Initial Data Syncronization page select Full and specify a shared network location. Note: if DB is already restored you do need to do full syncronization; in this case use “Joint Only”

21. Check Availability Group: In Object Explorer, expand AlwaysOn High Availability, then expand Availability Groups. You should now see the new Availability Group in this container. Right-click the Availability Group and click Show Dashboard.

22. In Failover Cluster Manager, click your cluster. Select Roles. The Availability Group name you used is a role on the cluster. That Availability Group does not have an IP address for client connections, because you did not configure a listener. You will configure the listener after you create an Azure load balancer.

23. Create Azure Load balancer with probe on 59999 and LB rule to load balance port 1433

24. Configure the listener

– Select the Networks node, and note the cluster network name. Use this name in the $ClusterNetworkName variable in the PowerShell script.

– Add the client access point.
a. The client access point is the network name that applications use to connect to the databases in an availability group. Create the client access point in Failover Cluster Manager.

a. Expand the cluster name, and then click Roles.

b. In the Roles pane, right-click the availability group name, and then select Add Resource > Client Access Point.

c. In the Name box, create a name for this new listener. The name for the new listener is the network name that applications use to connect to databases in the SQL Server availability group

d. To finish creating the listener, click Next twice, and then click Finish. Do not bring the listener or resource online at this point

– Configure IP resource for the Availability Group

a. Click the Resources tab, and then expand the client access point you created.
The client access point is offline.

b. Right-click the IP resource, and then click properties. Note the name of the IP address, and use it in the $IPResourceName variable in the PowerShell script.

c. Under IP Address, click Static IP Address. Set the IP address as the same address that you used when you set the load balancer address on the Azure portal.

– Make the SQL Server availability group resource dependent on the client access point.

a. In Failover Cluster Manager, click Roles, and then click your availability group.

b. On the Resources tab, under Other Resources, right-click the availability resource group, and then click Properties.

c. On the dependencies tab, add the name of the client access point (the listener) resource.

d. Click OK

– Make the client access point resource dependent on the IP address.

a. In Failover Cluster Manager, click Roles, and then click your availability group.

b. On the Resources tab, right-click the client access point resource under Server Name, and then click Properties.

c. Click the Dependencies tab. Verify that the IP address is a dependency. If it is not, set a dependency on the IP address. If there are multiple resources listed, verify that the IP addresses have OR, not AND, dependencies. Click OK.

d. Right-click the listener name, and then click Bring Online.

– Set the cluster parameters in PowerShell.

a. Copy the following PowerShell script to one of your SQL Server instances. Update the variables for your environment.

PowerShell

$ClusterNetworkName = “<MyClusterNetworkName>” # the cluster network name (Use Get-ClusterNetwork on Windows Server 2012 of higher to find the name)

$IPResourceName = “<IPResourceName>” # the IP Address resource name

$ILBIP = “<n.n.n.n>” # the IP Address of the Internal Load Balancer (ILB). This is the static IP address for the load balancer you configured in the Azure portal.

[int]$ProbePort = <nnnnn>

Import-Module FailoverClusters

Get-ClusterResource $IPResourceName | Set-ClusterParameter -Multiple @{“Address”=”$ILBIP”;”ProbePort”=$ProbePort;”SubnetMask”=”255.255.255.255″;”Network”=”$ClusterNetworkName”;”EnableDhcp”=0}

b. Set the cluster parameters by running the PowerShell script on one of the cluster nodes.

Note

If your SQL Server instances are in separate regions, you need to run the PowerShell script twice. The first time, use the $ILBIP and $ProbePort from the first region. The second time, use the $ILBIP and $ProbePort from the second region. The cluster network name and the cluster IP resource name are the same.

25. Set Listener Port

  1. Launch SQL Server Management Studio and connect to the primary replica.
  2. Navigate to AlwaysOn High Availability | Availability Groups | Availability Group Listeners.
  3. You should now see the listener name that you created in Failover Cluster Manager. Right-click the listener name and click Properties.
  4. In the Port box, specify the port number for the Availability Group listener by using the $EndpointPort you used earlier (1433 was the default), then click OK.

26. Test Connection to listener

For connectivity test use either telnet to port 1433 of listener or (from the server which is not the replica owner)

sqlcmd –S <listenername> -E

SCCM: Installation on hardened server

One of my Customers asked me to migrate an existing SCCM 2012 R2 to SCCM CB. They preferred side-by-side migration.

Everything looked good until I figure out the server they gave me for the new SCCM was hardened. I guess security team did it for good but as a result I had some fun with a trivial SCCM installation.

1. They used a third-party tool to remove TLS 1.0-1.1 and old SSL leavin only TLS 1.2 available. 3DES was killed too.

As a result, when I ran prereqchk.exe /Local before SCCM installation I received errors about SQL indexing, collation page (which I knew I set correctly), sysadmin membership etc… SQL looked good, but in

prereqchk log I saw: “Failed to connect to the SQL Server, connection type: SMS Master”

and in even log I observed: A fatal error occurred while creating a TLS client credential. The internal error state is 10013.

I removed fresh Machine keys from Programdata\microsoft\crypto\RSA – did not help

I set “Use FIPS compliant algorithms for encryption”  after that the error in event log changed saying TLS 1.0 protocol is using (never new it is FIPS compliant), but it is not configured.

So, at this point I ran IISCrypto and learnt the protocols are disabled.

As soon as I enabled the old obsolete TLS 1.0 prereqchk.exe passed smoothly and I started SCCM installation

Microsoft says  only SSL3.0 should be disabled and clearly requires both TLS 1.1 and 1.2 enabled. But in my case I still needed TLS 1.0 enabled too. So it looks like a working progress for me.

There is another article from Microsoft. It talks about TLS 1.2 configuration for SCCM CB 1610+. But it looks like it is about post-installation TLS 1.2 support and had an issue during installation. In addition, I tried my best to understand what should I configure on Windows Server 2016 with .Net 4.7 and SQL 2016 SP1 and as far as I understood I should do nothing, it supposed just work Smile. I would prefer Microsoft present the information in some kind of matrix for different .Net versions, OS versions and SQL…

2. Everything was fine until installer tried to setup a Management Point.

This time an error in ConfigMgrsetup.log said:

Unable to find an existing certificate in the store.  Creating a new self-signed certificate…    Configuration Manager Setup    11/20/2017 11:48:55 AM    3228 (0x0C9C)
Failed to release a handle to a cryptographic key (0x80070057)    Configuration Manager Setup    11/20/2017 11:48:55 AM    3228 (0x0C9C)
Failed to release a handle to a CSP or key container (0x80070057)    Configuration Manager Setup    11/20/2017 11:48:55 AM    3228 (0x0C9C)
Failed to create the certificate (0x8009000f)    Configuration Manager Setup    11/20/2017 11:48:55 AM    3228 (0x0C9C)
ERROR: Failed to find or create SQL Server certificate.    Configuration Manager Setup    11/20/2017 11:48:55 AM    3228 (0x0C9C)

this time I spent more time troubleshooting and finally opened a case with Microsoft. The tech found local Administrators was kicked out from permissions for Programdata\microsoft\crypto\RSA  and Setup could not create a private key there. We granted Full Control to local Administrators group, re-install MP and tis time it was setup.

3. Setup, bot not properly running – both standard tests (https://technet.microsoft.com/en-us/library/bb932118.aspx?f=255&MSPPError=-2147217396) from web browser gave me Internal Server Error (HTTP Error 500.19)

Fortunately I found Heinrich’s article (http://heinrichandsccm.blogspot.ca/2013/05/http-error-50019-internal-server-error.html). I re-installed WSUS  and MP started to work. After that I ran wsusutil for postinstall configuration and it finished successfully.

And after all changes above I succeeded to install SCM 4.0 before it failed with generic 1603 error.

USMT: Archive viewer

Mike Morawski wrote a nice wrapper for Migercover tool to browse .MIG files created by USMT/EasyTransfer. I am using that to check what exactly will be copied and tune up the config files.

The tool can be downloaded from Mike’s blog here: http://www.migee.com/2011/01/17/mig-recover-and-viewer-utility-alpha/

Please note, Mike is talking about USMT3/4 and the tool was not updated for a while. But it works for my USMT 10 at least for the current version 10.0.16299.15.

Azure: Clone VSTS Git repository to Visual Studio failed with error 400

I recently started to play with Azure Deployments via Visual Studio Team Services. The idea is having a source control for my ARM templates and keep my projects nice and tidy in one place.

I saw James Bannan presented on IT/DEV 2017 conference and liked this approach. Unfortunately, it is not clearly documented, or maybe I just cannot find a proper information since I am not a developer.

Anyway,  I integrated my Visual Studio 2017 with VSTS; that created a Git instance for me. From the VSTS portal I creted a new Project and tried to clone it to my VS2017. It miserably failed with Error 400.

Resolution: It turned out the clone process does not like spaces in project name :0 . fortunately there is a workaround I found here it describes a similar issue with cloning from tfs, but since the issue is actually on VS side it works for VSTS too. You basically need to cancel cloning in VS window and select “Clone Repository” from Project section. This will replace spaces in URL with %20 and in this case it finishes successfuly.

SCCM: Side-by-side migration. Changing Site for Clients.

I do site migrations for my Customers from time to time. Most often SCCM 2007/2012 to SCCM CB. The procedure is well documented and in generally works well. But the last step – move SCCM Clients from the old site to the new site is not very clear.

What I normally did is using Jason Sandys’ startup script (https://home.configmgrftw.com/configmgr-client-startup-script/)to re-install am SCCM Client with the new site code. But that always looked like a bit too much for me.

These days I am participating IT/DEV Connection conference and raised this scenario for panel of SCCM experts we have here. Here is the main points of the discussion:

1. Do not use SCCM ADMX template in GPO to change SCCM code only. Sometimes does not wrk, just breaks the client.

2. Jason’s script is probably the best way to re-point the client for today

3. Wally also mentioned we can create a package for a NEW client with the new site code on the OLD SCCM and deploy it. That will update the Client binaries and repoint it the same time.

4. Some people recommended to manage site assignment via site Boundaries (in this case we should separate boundaries for old and new SCCM). But as far as I remember it is not recommend even by MS itself.

So the bottom line is: after side-by-side migration we need to reinstall the Client using either Jason’s script above or method recommended by Wally.

Classic Azure AD Portal Access for CSP subscriptions

By default CSP users are OOL. But the Classic portal may be necessary. Here is a workaround: https://dirteam.com/sander/2017/07/03/creating-an-mfa-provider-when-you-have-csp-or-dreamspark/

PKI: Chrome 58 gives a warning with HTTPS connection to a Web Server

Scenario: Web Server protected by a certificate issued by Internal Certification Authority (Microsoft ADCS).

Certificate has a single Subject Name on it Internet Explorer works fine, only Chrome is affected

 

Warning:

This Server could not prove that it is <server name>; its security certificate is from [missing_subjectAltName]. This may be caused by a misconfiguration or an attacker intercepting your connection.

Resolution: I reissued the certificate and added Subject Alternative Name (SAN) with the same FQDN to it. After that I assigned the new certificate to the Web Server interface and restarted IIS. Chrome connects to the web server without any warnings now.

 

More about SAN: https://technet.microsoft.com/en-us/library/ff625722(v=ws.10).aspx

PKI: Enterprise PKI MMC displays a Subordinated CA as offline

I built a two layer PKI infrastructure and brought an Enterprise PKI MMC to verify the infrastructure health. All is ok but an Issuing CA. That was displayed with Status: ‘Error” and the message was “This CA is currently offline or unavailable”.

 

At the same time I could right-click it, select manage and it brought a very nice working CA MMC for me. So the CA is up and running and works fine, but for some reasons shown as offline in Enterprise PKI MMC.

Google did not bring too much, but search in Technet Forums gave a clue: https://social.technet.microsoft.com/Forums/en-US/fc8f6eba-447e-4e3f-a833-3b71bb3fc575/enterprise-pkiviewmsc-error-for-new-subca?forum=winserversecurity

I granted all permissions to my Domain Admins (this is Lab, otherwise it would be a custom security group). By default it was Manage CA and Issue and Manage Certificates only.

SNAGHTML48ce8f7f

and restarted the Certificate Services. After that Enterprise PKI became nice and green.

SCCM: SCUP 2011 on Windows Server 2016

found a nice post ( http://www.slr-corp.fr/2017/02/tips-tricks-installing-system-center-updates-publisher-scup-2011-windows-server-2016/) describing how to Install SCUP 2011 on Server 2016. Normally installation fails due to WSUS version check, but if you change WSUS version to  6.3.9600.1777 in registry (HKLM\SOFTWARE\Microsoft\Update Services\Server\Setup VersionString) for time of SCUP installation you can apparently finish SCUP installation. Of course without any support from MS Smile

or, as mentioned in the post comments just run SCUP .msi with /qb switch Winking smile

Meanwhile people continue to vote for this feature for SCCM CB here: https://configurationmanager.uservoice.com/forums/300492-ideas/suggestions/8803711-3rd-party-patching-scup-integration-with-sccm-co