Archive for the ‘Documentation’ Category

VMware ESXi and iSCSI storage issue.

I was having some trouble with a couple iSCSI volumes in a two member ESXi cluster. Two of the volumes were display in the client adapater details as mounted, but the volumes were not showing as mounted in the Storage pane or with a df command in the CLI.

Logs to check in for a possible hint:

# grep -i volumename vobd.log
# grep -i volumename vmkernel.log

When I listed the filesystems from the CLI using the following, the volumes were not listed or obviously mounted:

# esxcli storage filesystem list

List iSCSI adapters configured:

# esxcli iscsi adapter list

A rescan of all the adapters did not work either:
Rescan adapter:

# esxcli storage core adapter rescan -a

To list all the devices and their world IDs:

# esxcli storage core device world list

To only list the world IDs tied to one device:
# esxcli storage core device world list -d mydeviceid

List all guests and their World ID:

# esxcli vm process list

Restart management services:

# services.sh restart

The vmkernel.log had a clue. A message stating that “Device mydeviceid detected to be a snapshot:”

# grep -i mydeviceid vmkernel.log

Listing the snapshots revealed the issue:
List snapshot:

# esxcli storage vmfs snapshot list

Both of my troubled volumes were listed as snapshots. I was told by VMware that this can happen if something changes in the META data of the iSCSI SAN/NAS. This was possible in my case, because I just updated my FreeNAS to the latest version. All that needed to be done was remove the snapshots. Once removed the volumes were mounted immediately.
Remove snapshot:

# esxcli storage vmfs snapshot mount -u “59b153b3-86f464ec-999d-a0d3c1f0cdf0”
# esxcli storage vmfs snapshot mount -u “59b1a680-bc18c507-831a-2c768a56eb24”

User Specific ssh/sftp/scp Customizations in CentOS.

I ran into a situation where I was trying to place files for support, and they only supported 3des ciphers (3des-cbc,blowfish-cbc,3des-cbc). The global ssh client configuration on my system only supported aes ciphers. Instead of adding the 3des to the global configuration (/etc/ssh/ssh_config), I wanted to add it to just one account.

$ vi ~/.ssh/config
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,3des-cbc

$ chmod 400 ~/.ssh/config

Then, I could run ssh/sftp/scp with -vvv to verify. You should see the following output:


debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,3des-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,3des-cbc

Of course, you can add any customization you want to the ~/.ssh/config file you want. This is just an example. What got me was the global config file is call ssh_config, while the user config file is called config. man ssh_config help me discover the correct name. If you do not have the correct name, you need to pass a -F myspecialcustomconfigfile to the ssh/sftp/scp command.

Modify Exchange Message Size Limits

There are a number ways to control message size in Exchange. Many actually. Even down to the header size.

Reference: https://technet.microsoft.com/en-us/library/bb124345(v=exchg.141).aspx

Here are the options I have been most likely to use:
This is where you set your server parameters. This setting applies to all users on the server:
To view your current settings:

Get-TransportConfig | Select MaxSendSize,MaxReceiveSize

To modify your settings:

Set-TransportConfig -MaxSendSize 15MB -MaxReceiveSize 15MB

Here is where you can set the parameter based on a receive connector only:

Get-RecieveConnector | Select MaxMessageSize

To modify:

Set-ReceiveConnector -Identity “receiveconnectorname” -MaxMessageSize 15MB

And this for a send connector:

Get-SendConnector | Select MaxMessageSize

To modify:

Set-SendConnector -Identity “sendconnectorname” -MaxMessageSize 15MB

This is where you would set it for the individual mailbox. So, you might want have your other settings high, and restrict by mailbox if the need is there.
To view a mailbox’s limits:

Get-Mailbox | Select Name,MaxSendSize,MaxReceiveSize
Get-Mailbox mailboxname | Select Name,MaxSendSize,MaxReceiveSize

To modify:

Set-Mailbox jgz -MaxSendSize 100MB -MaxReceiveSize 100MB

How To Put an iPhone 8/Plus Into Recovery Mode

To put a iPhone 8 or 8 Plus into recovery mode is a little different than in past models.

Have your phone plugged into iTunes and turned off.

The following steps should be be done quickly:

1) Press and release the Volume Up button.

2) Press and release the Volume Down button.

3) Press and hold the Power button on the side of the phone. Keep holding it until to iTunes tell you the phone is in recovery mode.

Windows 2012 R2 – seize roles from failed domain controller.

I had to deal with a really neglected domain, and found that all the FSMO roles were on a domain controller that no longer functioned or existed. I had to get the roles on the working server. Using convental methods in the UI or the ntdsutil to transfer the roles succeeded. I had no choice, but to seize all the roles from the missing server. All of these tasks were completed on the domain controller I wanted the roles on as the domain\Administrator.

Check the current roles holders:

C:\>netdom query fsmo
Schema master MYOLDDC1.mydomain.local
Domain naming master MYOLDDC1.mydomain.local
PDC MYOLDDC1.mydomain.local
RID pool manager MYOLDDC1.mydomain.local
Infrastructure master MYOLDDC1.mydomain.local
The command completed successfully.

Enter the ntdsutil utility by entering ntdsutil:

C:\>ntdsutil

And then roles:

ntdsutil: roles

You see the options by entering a question mark at the “fsmo maintenance” prompt. Obviously, this where you also transfer the roles if possible (not so in my case):

fsmo maintenance: ?

? – Show this help information
Connections – Connect to a specific AD DC/LDS instance
Help – Show this help information
Quit – Return to the prior menu
Seize infrastructure master – Overwrite infrastructure role on connected server
Seize naming master – Overwrite Naming Master role on connected server
Seize PDC – Overwrite PDC role on connected server
Seize RID master – Overwrite RID role on connected server
Seize schema master – Overwrite schema role on connected server
Select operation target – Select sites, servers, domains, roles and
naming contexts
Transfer infrastructure master – Make connected server the infrastructure master
Transfer naming master – Make connected server the naming master
Transfer PDC – Make connected server the PDC
Transfer RID master – Make connected server the RID master
Transfer schema master – Make connected server the schema master

Seize the roles one at a time. Each takes a while to complete, but they do.:

fsmo maintenance: seize pdc
Attempting safe transfer of PDC FSMO before seizure.
ldap_modify_sW error 0x34(52 (Unavailable).
Ldap extended error message is 000020AF: SvcErr: DSID-03210617, problem 5002 (UNAVAILABLE), data 1722

Win32 error returned is 0x20af(The requested FSMO operation failed. The current FSMO holder could not be contacted.)
)
Depending on the error code this may indicate a connection,
ldap, or role transfer error.
Transfer of PDC FSMO failed, proceeding with seizure …
Server “mydc01” knows about 5 roles
Schema – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Naming Master – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
PDC – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
RID – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Infrastructure – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
fsmo maintenance:
fsmo maintenance: seize naming master
Attempting safe transfer of domain naming FSMO before seizure.
ldap_modify_sW error 0x34(52 (Unavailable).
Ldap extended error message is 000020AF: SvcErr: DSID-0321041F, problem 5002 (UNAVAILABLE), data 1722

Win32 error returned is 0x20af(The requested FSMO operation failed. The current FSMO holder could not be contacted.)
)
Depending on the error code this may indicate a connection,ldap, or role transfer error.
Transfer of domain naming FSMO failed, proceeding with seizure …
Server “mydc01” knows about 5 roles
Schema – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Naming Master – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
PDC – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
RID – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Infrastructure – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
fsmo maintenance: seize rid master
Attempting safe transfer of RID FSMO before seizure.
ldap_modify_sW error 0x34(52 (Unavailable).
Ldap extended error message is 000020AF: SvcErr: DSID-03210F70, problem 5002 (UNAVAILABLE), data 1722

Win32 error returned is 0x20af(The requested FSMO operation failed. The current FSMO holder could not be contacted.)
)
Depending on the error code this may indicate a connection, ldap, or role transfer error.
Transfer of RID FSMO failed, proceeding with seizure …
Searching for highest rid pool in domain
Server “mydc01” knows about 5 roles
Schema – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Naming Master – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
PDC – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
RID – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Infrastructure – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
fsmo maintenance: seize schema master
Attempting safe transfer of schema FSMO before seizure.
ldap_modify_sW error 0x34(52 (Unavailable).
Ldap extended error message is 000020AF: SvcErr: DSID-0321041F, problem 5002 (UNAVAILABLE), data 1722

Win32 error returned is 0x20af(The requested FSMO operation failed. The current FSMO holder could not be contacted.)
)
Depending on the error code this may indicate a connection, ldap, or role transfer error.
Transfer of schema FSMO failed, proceeding with seizure …
Server “mydc01” knows about 5 roles
Schema – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Naming Master – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
PDC – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
RID – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Infrastructure – CN=NTDS Settings,CN=MYOLDDC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
fsmo maintenance: seize infrastructure master
Attempting safe transfer of infrastructure FSMO before seizure.
ldap_modify_sW error 0x34(52 (Unavailable).
Ldap extended error message is 000020AF: SvcErr: DSID-0321041F, problem 5002 (UNAVAILABLE), data 1722

Win32 error returned is 0x20af(The requested FSMO operation failed. The current FSMO holder could not be contacted.)
)
Depending on the error code this may indicate a connection, ldap, or role transfer error.
Transfer of infrastructure FSMO failed, proceeding with seizure …
Server “mydc01” knows about 5 roles
Schema – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Naming Master – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
PDC – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
RID – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
Infrastructure – CN=NTDS Settings,CN=MYDC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mydomain,DC=local
fsmo maintenance:

Check to role locations to verify using netdom again:

C:\>netdom query fsmo
Schema master MYDC01.mydomain.local
Domain naming master MYDC01.mydomain.local
PDC MYDC01.mydomain.local
RID pool manager MYDC01.mydomain.local
Infrastructure master MYDC01.mydomain.local
The command completed successfully.

Cisco 1142 AP won’t join after WLC reboot.

Update: It looks like it the end of the line for 2106, because there is no update and hasn’t been since 2015 which I have installed.

I have a couple Cisco LAP1142N access points and a Cisco WLC2106. I noticed some pretty consistent packet loss on the management interface of the WLC. I opted to reload the WLC, since it had been up a long time, to see if it would help. However, when it came up and the access points attempted to join the WLC, I was getting certificate errors like these:

*Aug 30 18:17:08.097: %LWAPP-3-CLIENTERRORLOG: Peer certificate verification failed
*Aug 30 18:17:08.097: %CAPWAP-3-ERRORLOG: Certificate verification failed!
*Aug 30 18:17:08.097: DTLS_CLIENT_ERROR: ../capwap/capwap_wtp_dtls.c:352 Certificate verified failed!
*Aug 30 18:17:08.097: %DTLS-4-BAD_CERT: Certificate verification failed. Peer IP: 192.168.0.141
*Aug 30 18:17:08.097: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 192.168.0.141:5246
*Aug 30 18:17:08.097: %DTLS-3-BAD_RECORD: Erroneous record received from 192.168.0.141: Malformed Certificate
*Aug 30 18:17:08.097: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.0.141:5246
*Aug 30 18:17:08.098: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 3 combination.
*Aug 30 18:17:08.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.0.141 peer_port: 5246
*Aug 30 18:17:08.095: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed. The certificate (SN: myserialnumber) has expired. Validity period ended on 20:42:36 UTC Aug 18 2017
*Aug 30 18:17:08.096: %LWAPP-3-CLIENTERRORLOG: Peer certificate verification failed
*Aug 30 18:17:08.097: %CAPWAP-3-ERRORLOG: Certificate verification failed!
*Aug 30 18:17:08.097: DTLS_CLIENT_ERROR: ../capwap/capwap_wtp_dtls.c:352 Certificate verified failed!
*Aug 30 18:17:08.097: %DTLS-4-BAD_CERT: Certificate verification failed. Peer IP: 192.168.0.141
*Aug 30 18:17:08.097: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 192.168.0.141:5246
*Aug 30 18:17:08.097: %DTLS-3-BAD_RECORD: Erroneous record received from 192.168.0.141: Malformed Certificate
*Aug 30 18:17:08.097: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.0.141:5246
*Aug 30 18:17:08.098: %CAPWAP-3-ERRORLOG: Invalid event 38 & state 3 combination.
*Aug 30 18:17:08.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 192.168.0.141 peer_port: 5246

I checked the time on all of the devices. My WLC is synced to my internal NTP server, and the access points were syncing their time with the WLC when they would load. I found the following field notice from Cisco that addresses the issue:

Field Notice: FN – 63942 – Wireless Lightweight Access Points and WLAN Controllers Fail to Create CAPWAP/LWAPP Connections Due to Certificate Expiration

I applied the workaround, since I currently do not have the software upgrade:

(Cisco Controller) config>ap lifetime-check mic enable

(Cisco Controller) config>ap lifetime-check ssc enable

(Cisco Controller) config>exit
(Cisco Controller) >save
(Cisco Controller) save>config

Are you sure you want to save? (y/n) y

Configuration Saved!

I reloaded one of the access points, but by the time it came up and joined, I noticed that the other access point had already joined, so I guess I didn’t need to do that.

Enable Windows Server To Utilize Invoke-Command Remotely

When I attempted to the Powershell option Invoke-Command against an old server, I was getting the following:

Connecting to remote server servername failed with the following error message : The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests.
Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure
the WinRM service: “winrm quickconfig”. For more information, see the about_Remote_Troubleshooting Help topic.

Fortunately, it told me what to do resolve the issue. Nice:

C:\>winrm quickconfig
WinRM already is set up to receive requests on this machine.
WinRM is not set up to allow remote access to this machine for management.
The following changes must be made:

Create a WinRM listener on HTTP://* to accept WS-Man requests to any IP on this
machine.

Make these changes [y/n]? y

WinRM has been updated for remote management.

Created a WinRM listener on HTTP://* to accept WS-Man requests to any IP on this
machine.

Cisco 1941 password recovery

Note: This procedure is applicable to most Cisco routers, however the key is knowing the register to use.

Turn the power off.
Turn the power on.
About when you see the following message, hit Ctrl-Break (yes the Pause/Break key).

Readonly ROMMON initialized

You should be presented with the following prompt:

rommon 1 >

Enter confreg 0x2142:

rommon 1 > confreg 0x2142

Then, you will get the following message:

You must reset or power cycle for new config to take effect

Enter reset:

rommon 2 > reset

The router will reboot and start the initial configuration wizard. Just say “No” to skip. This will drop you to a “Router>” prompt.

Enter enable, and you will presented with a “Router#” prompt.

Copy your startup-config to running-config (make sure you do not switch the order or you will lose your configuration):

Router#copy startup-config running-config

Then reset the password (I set it to “cisco” below.):

Router#configure term
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#enable secret cisco

Then, type the following:

config-register 0x2142

If you cannot remember the register number from earlier, you can find by issuing the following:

Router(config)#do show version

Enter:

Router(config)#end

And save:

Router#write mem
Building configuration…
[OK]

Then reload to test:

Router#reload
Proceed with reload? [confirm]

Awk and cases

Good stuff here. I always like to pick up these little things along the way.

If you want to change the case of a string using awk:
Lower case:

$ echo myuppercasestring | awk ‘{print tolower($1)}’

Upper case:

$ echo mylowercasestring | awk ‘{print toupper($1)}’

I used something like this to create little of commands to rename a bunch of upper case file names to lower case file names:

$ ls -c1 | awk ‘{print “mv ” $1 ” ” tolower($1)}’

CentOS 7 – package conflict during update.

I was having trouble getting the most recently installed kernel to boot (not the latest release in the repository). It just immediately crashed like it was a grub issue. So, I decided to update the server to an even later kernel, since it is not really a production server.

However, when I did I kept getting the following conflict message:

Error: kernel conflicts with kmod-20-8.el7_2.x86_64

This what took care of the issue for me:

After running this command, I discovered that it was not an issue with an incomplete installation during my last updates.

# yum-complete-transaction –cleanup-only

Then, I ran the following, which removed a lot of duplicate packages:

package-cleanup –cleandupes

Then, I updated the server again:

# yum -y update

Rebooted the latest kernel in the repository without any issues.

Return top

INFORMATION