• On October 29, 2018
  • By

Day 0: Virtual Environment Setup, Scripts, and WinDbg

Overview

Today is the day you begin your wild journey into hypervisor development. It’s going to be miserable, rewarding, frustrating, and exciting – all at the same time. In this article, we accelerate head on into setting up our virtual environment using VMware (taking advantage of their nested virtualization), write scripts for quick stand-up and shutdown of the virtual environment and test files, and configure WinDbg to automatically connect and break-in to our test environment when loaded. We’ll also setup a workspace and walk through the two most utilized methods of kernel debugging a virtual machine (network, and serial.) I’ll put a resource at the end if you find yourself wanting to use VirtualKd to improve your kernel debugging performance on VMware – although, I will go ahead and mention that I’ve not used VirtualKd when debugging a hypervisor on VMware so I’m unsure if there will be any issues. Any and all issues you experience are welcome to be reported to me, so that I can give readers the heads up or solutions to those issues.

Disclaimer: If you didn’t read the introduction to the series this hypervisor is Intel based taking advantage of VT-x. This series is written with more experienced developers in mind, those who have some experience with x86/x86-64 assembly, C, system programming, and/or driver development. If you’re new to x86/x86-64 assembly, C, or the fundamentals of the Intel architecture then strap in because it’s going to be a wild and incredibly brutal ride. However, if you’re up for the challenge and willing to push yourself to learn I have no doubts you can keep up.

Note: This article is very verbose and does extrapolate on somewhat trivial things because I want to make sure the reader is operating in a near exact environment as when this was developed to reduce the possibility of errors.

For this series you will need the following installed on your development machine:

Make sure you have those items installed and all the debugging tools associated with WDK 10.

Let’s dig in ladies and gentlemen.

Virtual Environment Setup/Configuration

In this section we’re going to be installing the proper operating system, making sure it functions properly, configuring the test environment, downloading and testing the tools required for debugging, and preparing the test machine for smooth interaction when lots of testing will be required later. There’s a specific configuration we’re going to follow until Day 3 when we begin multi-processor initialization. Please don’t skip any steps, and make sure the hardware configuration is the same as mine for your virtual environment.

For this series, we’ll be using VMware Workstation 15 Professional and will be running Windows 10 Professional Version 1803. If you’re looking for an ISO download for Windows 10 Professional and have a key, check this link. Once you’ve acquired a copy, continue reading below for setting up the virtual machine and the hardware configuration we’ll be using.

To get started, you’re going to create a new virtual machine (if you don’t already have one setup) using the Typical option as shown below.

 

 

Proceed forward and browse to where the Windows 10 Professional ISO is located. Select it and click next, enter your product key, name your virtual machine, and choose a location for all the VM files. Once that’s complete, leave the default disk size at 60 GB, and split the virtual disk into multiple files. Now we’re going to make an important change that won’t be modified until Day 3.

You’re going to click on ‘Customize Hardware‘, and navigate to the Processors selection, once selected you need to change whatever the current setting is (default should be 1 processor, 2 cores) to the following:

And check ‘Virtualize Intel VT-x/EPT or AMD-V/RVI‘ as pictured below.

Once this is done, click close, and then finish in the wizard, and let the VM start and operating system install. Your virtual machine is now setup, but the real configuration begins after the OS is installed.

Operating System Configuration

Once the operating system has finished installation, we need to shut down some services, disable some features, edit the boot configuration, and setup shared folders. Make sure you follow each configuration change because missing one could result in a conflict during execution of the hypervisor. First, we need to shut down some services. These services utilize a lot of CPU and can slow down rebooting/startup of the virtual machine as well as cause a lot of noise on debug logs when monitoring global kernel events (I’ll cover that later with WinDbg.)

 

— Disable Windows Defender

Windows Defender is a great addition and pre-installed protection software for Windows 10 (much better than previous years), however, it can slow down execution and at least during my use was constantly delivering notifications about updates and other security features that I wanted to disable to maximize the utilization of my playground. For lack of any other explanation, I disabled it because it was annoying me and invading my screen space (which is limited on a VM’s display anyways.)

To disable Windows Defender we’re going to use gpedit. This is a Windows module that is used to edit group policies for the system. We’re going to specifically modify the Windows Defender Antivirus policy. To do this, first press ‘Windows + R‘ to open the run dialog, type gpedit.msc (the .msc is required) and hit enter. Once the dialog is open follow the screenshots below.

First, navigate through ‘Computer Configuration‘ to ‘Administrative Templates‘, once that drop down is rendered find ‘Windows Components‘.

 

 

Next, navigate into ‘Windows Components‘ and find ‘Windows Defender Antivirus‘. In the main frame you should see a policy titled ‘Turn off Windows Defender Antivirus‘, click it and change the setting from ‘Not Configured‘ to ‘Enabled‘. Yes, the enabled setting actually disables the antivirus.

 

 

Good, now you’ve successfully disabled Windows Defender and we can breeze through the rest of the disables.

 

— Disable UAC (User Account Control)

This is incredibly simple to disable. Press the Windows key, and type ‘User Account Control‘. A suggestion for modifying User Account Control settings should appear, click on it and a dialog should appear with a vertical slider. Slide it all the way down, we don’t want constant notifications when/if we try to run things as administrator. We supposedly know what we’re doing…

 

 

— Disable Hyper-V, and Virtualization Based Security

To avoid conflicts when testing our hypervisor we want to disable the Windows Hypervisor. This can be done through gpedit, except this time you’ll navigate to ‘System‘ under ‘Administrative Templates‘ and select the ‘Turn On Virtualization Based Security‘ option, then select ‘Disabled‘.

Disabling Hyper-V is trivial as well, press the Windows key to open the search dialog, type ‘Windows features‘ and select the ‘Turn Windows features on or off‘ suggestion.

 

 

Once open, navigate to Hyper-V and uncheck the box. Click OK, and we’re done.

Once these three are disabled you’ll need to reboot your VM before proceeding the next part.

Operating System Configuration Continued

In this subsection we’re going to start with downloading and setting up our tools, then modify our boot configuration to allow for unsigned drivers to be loaded as well as kernel debugging, followed by setting up shared folders. The two methods of kernel debugging (network, and/or serial) will be detailed. I highly recommend network debugging through VMware because the debugger latency is substantially reduced. If you’d rather take the easy way out, use serial debugging. Quick, and painless (until the work starts.)

In the overview, I mentioned the necessity for SysInternals and if you forgot, or didn’t download it now is the time because we’re going to use some very valuable tools from the suite. Once downloaded, you’ll want to pull Process Explorer and DebugView from the archive and put them on your VM’s desktop. Process Explorer doesn’t require any setup, but because DebugView can be finicky we’re going to play it safe and pre-configure it before any development occurs.

 

— DebugView

DebugView is a tool that allows the user to monitor debug output on the local computer, or a remote system so long as you have access over TCP/IP. One of its capabilities, the one we’ll be using throughout this series, is the monitoring of kernel debug output. It doesn’t require any special tools, no debuggers, or otherwise; only the use of DbgPrintEx, DbgPrint, or OutputDebugString. The only issues we’ll be covering regarding DebugView setup is that it can be temperamental with debug output – that is, it will sometimes not print debug output for device drivers or applications. We’re going to add our own registry key, create a simple project (it should be the one you’ll be using throughout this series), and write a test key to ensure output is showing.

Other known issues with DebugView:

  • Driver in use errors upon closing and reopening.
  • Not printing debug output.
  • Printing other event logs to the view after disabling via filter.

The first issue is handled by restarting the VM, as obnoxious as it is. The last two have solutions provided below and it is recommended that these be implemented prior to use to save time. To ensure that debug output is produced and displayed in DebugView we’re going to add our own registry key to our system control set, as well as modify the boot configuration to enable debugging.

You’ll need to open regedit, and navigate to the registry tree HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager. After locating, right click and create a new Key called Debug Print Filter. Select the new key, right click in the value view pane and create a new DWORD value named IHVDRIVER with the value 0xFFFF.

As an aside, this registry modification is only required if Enable Verbose Kernel Output is disabled. Otherwise, all debug output will be displayed in DebugView. Your new registry entry should look like this:

 

 

Following this, right click the start button and open a new adminstrator CMD. Enter bcdedit /debug ON and hit enter. The operation should have completed successfully. Next execute bcdedit /set testsigning on – this allows us to load unsigned drivers. All that is required now is to restart the VM and resume configuration. After your reboot we’re going to setup a project, ensure that DebugView is working properly, and then begin scripting and setting up our project workspace in WinDbg.

 

— Project Setup and Debug Output

It’s time to create our project and write some test code. I’m going to quickly summarize what project type and what I did to test that everything was working prior to development. If you’re new to driver basics/development, please refer to the recommended reading section at the end of the article.

At this point you should have Windows Driver Kit installed and working in Visual Studio. Assuming this, I use an empty WDM driver as the project type, and create with the default settings. Upon creation you should create an entrypoint source file and create a DriverEntry.

#include <ntddk.h>

NTSTATUS
DriverEntry(
  PDRIVER_OBJECT DriverObject,
  PUNICODE_STRING RegistryPath
)
{
  UNREFERENCED_PARAMETER( DriverObject );
  UNREFERENCED_PARAMETER( RegistryPath );

  DbgPrint( "%s - Output Test from ReHV.\n", __FUNCTION__ );

  return STATUS_SUCCESS;
}

Embedded above is the DriverEntry definition I used when testing the project. DbgPrint is the API that will be outputting the information above in DebugView. Prior to creating the shared folder between systems be sure to save and build your project so that the build folder is available for selection. Once done, we’re going to setup shared folders, build, perform our first init test on the VM, and finish up with setting up our kernel debugger. Shared folders will be incredibly useful for testing speed and efficiency, as well as used in our scripts later on. To start, we’re going to configure the build shared folder.

Creating and enabling a shared folder on VMware is simple. Once the virtual machine is created it can be modified by selecting the desired VM, clicking Edit virtual machine settings, and under the settings dialog select the Options tab. This is depicted below.

After navigating to the correct area, on the right side pane make sure your settings are the same as shown below.

After enabling these settings, go to the lower control pane and select Add… This is where you will browse to your project folder where the project will be built and allow a mapping from your local system to the VM. This makes it so that every time you clean/build your project it will be updated in your VM and save time versus dragging and dropping the newest build and starting the service.

Once the shared folder is setup, power on your VM. Locate your shared folder by opening a new File Explorer. A network directory with the name vmware-host should be visible. Inside of here is where your shared folders are created, find where your build folder is. The path should look like \\vmware-host\Shared Folders\<project name>. This is the path that will be used when you create your service using the Service Control Manager and for use in the Batch script for creating the service. The contents of this folder should include the security catalog (.cat), setup information (.inf), and the driver system file (.sys). If these contents are not in this folder you’ll need to remap your Shared Folder host path, or navigate to the correct folder and save the path.

Now that we’ve complete shared folder mapping, let’s open DebugView, setup exclusions, and create a test service to make sure the output is working. Once DebugView is open, navigate to the capture drop down, and select the following controls:

Once done, open an administrator command prompt. We’re going to use the Service Control Manager to create and start our service and see our test print. To do so, we need to create our service by typing the following command: sc create <service name> binPath="\\vmware-host\Shared Folders\<project name>.sys" type=kernel. After pressing enter you should have been greeted with a success message.

At this point DebugView should be open and properly configured to capture kernel debug output. To start our service and test that our debug output is working we will type sc start <service name>, in my case it is rehv. After pressing enter, information regarding the service status should appear with the state value 4 (running). Bring DebugView back into the foreground and look at the debug display. You should see your DriverEntry debug output displayed in the debug log.

That’s it. Our debug output is working and we have our shared folders setup. We’re ready to dive into some scripting to automate the starting and stopping of our service.

Scripting

Since we’re going to be starting and stopping our service to test different components of the hypervisor it’s important to make starting a stopping as quick as possible. This is made easy through the use of batch. Some prefer the use of PowerShell scripts, handling errors, and resetting the service from in the batch file. However, we’re not going to run into those issues in these articles – if you’re wanting to beef up your approach just be ready to spend a little bit longer on the scripting portion of setup.

This is going to be quick and painless. To start we want to open up your text editor of choice. We’re going to simplify the start up and tear down of the service to prevent us from having to open an administrator command prompt every time we want to start or stop the service. Below is the content of my start and stop batch files for the service, they will have to be run as administrator for the commands to succeed.

— rehv_start.bat

sc create rehv binPath="\\vmware-host\Shared Folders\rehv.sys" type=kernel
sc start rehv

— rehv_stop.bat

sc stop rehv
sc delete rehv

Quick, simple, effective. You’ll need to substitute your own project parameters in the commands. To make this a single file ordeal you could employ the use of Global System Variables to determine whether the service was started or stopped properly, and perform error checking / handling using these variables in the script. I don’t want to unnecessarily over-complicate things, however the resource is provided in the highlighted keyword should you be interested in implementing this yourself.

Note: The stop argument will hang if no unload or device creation/deletion takes place because of a reference to an object that can not be dereferenced until system reset. This can be seen by using ObGetObjectPointerCount to get a reference count on the object in question. To prevent this from happening we will implement an unload routine and create/destroy a device in their respective functions in the next article.

Kernel Debugging w/ WinDbg

In this section we’re going to discuss setting up a workspace. The setup of kernel debugging on VMware is a topic that is written about in various places, however, I recommend following the guide provided here if you haven’t setup kernel debugging already. In this series we use serial debugging, despite it’s drawbacks. It is important to be compatible with all readers and not the ones who have NICs that support it. I’ll also recommend that instead of manually connecting and breaking into the VM every time you want to test something that you take a look at the last section of the article previously mentioned. It explains a shortcut that allows you to automatically begin kernel debugging the desired named pipe without having to use any environment variable hacks.

The important part of this section is the use of a workspace. A workspace is vital in debugging because it saves you from having to readjust and remodel your view every time you want to perform kernel debugging. More specifically, we’ll be setting up a default kernel-mode workspace.

After setting up your kernel debugging named pipe, open up WinDbg and (hopefully automatically) connect. Go ahead and break in by pressing Control+Break. Once DbgBreakPointWithStatus is executed we’re going to setup our workspace. This is done by using the command bar with window shortcuts.

I recommend for this type of development and testing to have Source mode enabled. As well as using the following windows:

  1. Registers
  2. Locals
  3. Call Stack
  4. Command Window

You can organize these however you’d like, it’s important however that you have these available when your debugging session begins. You’ll be dealing with a good amount of custom assembly and error tracking as well as verifying that stacks are properly aligned prior to executing certain instructions. This is what my layout looks like for a simple debugging session such as what we’ll be doing throughout development.

After organizing your workspace to your liking, find the File menu, and locate Save Workspace As… The default should be AMD64, you can name it whatever you’d like. I kept it as AMD64 for this project. Click OK and navigate back through the File menu except this time click Save Workspace. After following the guide on triplefault.io and setting up a workspace, whenever you launch the WinDbg from the shortcut associated with this project it should automatically connect, and load your workspace upon breaking in.

If you have trouble with the workspace not automatically loading once the debug session begins, right click on your shortcut and append -W <workspace name> to the Target field. This will force load the workspace layout at start of WinDbg using that shortcut. And that’s it. You’ve successfully setup your virtual environment for testing and development of your basic hypervisor. The rest of the articles will be over the technical details, implementation, and testing of a type-2 hypervisor.

Conclusion

In this article, we’ve gone through to ensure that the reader and the authors environment are setup the same way to minimize conflicts and errors that may happen. It’s a routine setup for those familiar, but those who are inexperienced with proper driver development or debugging may find it helpful to have everything put together ahead of time. If you’ve read all the way through and followed along, you’re ready for tomorrow – when the real development and unit tests begin. The next article will be much more in depth and require a lot of supplemental reading on your part.

As mentioned in the introduction, if you’re interested in using a fast and reliable method of kernel debugging on VMware, I recommend using VirtualKd. This is the link to the installation instructions.

If you’re interested in tightening up your debugging tools and scripts follow the references in the content above or see the non-comprehensive list below. If you’re looking to get ahead of the article tomorrow see the recommended reading list below!

Non-Comprehensive Resources for Day 0

 

Recommended Reading/Review For Day 1

As always, your questions, comments, or general feedback is appreciated. See you tomorrow!

Author

Tags: , , , , , , , , ,

4 thoughts on “Day 0: Virtual Environment Setup, Scripts, and WinDbg”

Leave a Reply