I was tasked with building a test server so that a client could test a new feature in a hotfix that was going to be applied to their software package. The server is installed with software and configured to look like a mirror of their production box, right down to the hardware on which the software runs and I’ve done this many times before so I didn’t think this was going to be a huge problem. As soon as that sentence entered my thoughts, I should have known better.
Once the hardware was secured and the OS (Windows Server 2003 SP2) was installed, it was time to get about the business of making this look like the client’s production environment. Two of the tasks involved in this were…
- Installing Citrix so that the client could access the box for testing, and
- Adding system DSN’s via the ODBC administrator so that data could be pulled from SQL server to the locally installed database application (to be installed later).
Citrix came first. Once that was completed, I opened the ODBC administrator and found to my horror that when I either tried to create a new system DSN, the ODBC administrator shut down the second you hit the configure button. I won’t bore you with the details, but this led to myself and the intrepid I.T. staff I work with running down several plausible, but incorrect and time-wasting paths (A missed Windows software update? Bad disk cluster? Bad OS install? Me being the pawn of a vengeful supreme being’s sick sense of humor?). Fortunately, as with so many other things, the interwebs came to our rescue because huddled in an obscure little corner of the web was the solution to our problem.
Once the frustration was vented and our thoughts were collected, we actually did something we should have done from the beginning: looked at the event logs from when the ODBC administrator failures occurred. We found this stream of jibberish:
0000: 41 70 70 6c 69 63 61 74 Applicat
0008: 69 6f 6e 20 46 61 69 6c ion Fail
0010: 75 72 65 20 20 6f 64 62 ure odb
0018: 63 61 64 33 32 2e 65 78 cad32.ex
0020: 65 20 33 2e 35 32 36 2e e 3.526.
0028: 33 39 35 39 2e 30 20 69 3959.0 i
0030: 6e 20 75 6e 6b 6e 6f 77 n unknow
0038: 6e 20 30 2e 30 2e 30 2e n 0.0.0.
0040: 30 20 61 74 20 6f 66 66 0 at off
0048: 73 65 74 20 35 62 62 33 set 5bb3
0050: 31 32 64 65 12de
The key to this mess is this snippet: 5bb312de
This is the address in memory where the access violation is created by the Citrix Virtual Memory Optimization service. In order to not see this again, you simply turn off that service.
I was trying to upgrade an application, and due to the list of complexities involved in the upgrade process, I had decided that the best course of action was to back up all crtical data, uninstall the previous version, and then do a clean install of the new version.
Having done this before on the other production boxes we have, and this box being the same model, OS, and Citrix install as the other boxes, I made a horrible mistake… I assumed that it would be the same smooth process as had occurred on the other servers.
So I grab my word doc chock full o’ meticulous notes on how to upgrade this monster, and follow each step to the letter and after about 10 minutes or so, the uninstall goes smooth as silk. I go to do the re-install and I get this message upon clicking the first executable file:
Windows cannot access the specified device, path, or file. You may not have the appropriate permissions to access the item.
Upon seeing this, I start to go through in my head the list of usual offenders.
- Do I have administrative rights? Check.
- Since this box runs Citrix, did I disable logons for other users and enable installation mode? Check.
- Anything set as read-only that shouldn’t be? Nope.
At this point, I was damn near befuddled, and when you get befuddled, you theories start getting more and more far fetched. Starting with “Is my account corrupt?” then after logging on using another admin account and getting the same result, you migrate to “Have the Chinese taken control of my server and they are just toying with me right now?” and culminating with “Am I really doing this or am I just another hairless pig fetus floating in a plexiglass bathtub somewhere deep inside inside the Matrix?”
It was at this point that I decided to call Saulo, one of our intrepid Windows admins where I work. Working with Saulo is great for several reasons… you can get him on weekends, he knows his shit, etc…. but his specialty in this case is that he can find solutions when all of your mere mortal options have been exhausted. After some trial and error and research, he found this little nugget of wisdom that I will now impart to you:
- Open Internet Explorer (I know. Seems totally unrelated, but this is Windows, so it’s not supposed to be intuitive. Just work with me here.)
- Go to Tools -> Internet Options.
- Select the Security tab
- Select Internet and then click the Custom Level button.
- In the Miscellaneous section, the Launching applications and unsafe files setting was set to Disable, which is the root cause of my problem. To fix this, set it to Prompt (recommended).
This screenshot will show the visual details of the steps shown above.