VMware Virtual Security and Compliance

Monday, June 22, 2009

In my prior blog post I discussed the richness of our support for securing heterogeneous virtualization platforms. In this blog post I want to drill down in detail on some of the value we provide in terms of delivering identity and access management to VMware environments.

Before I got into details, I want to point a few great resources:

A new video chalktalk on integrating VMware ESX with Active Directory.
An amazing and detailed 30 page whitepaper on our support for securing VMware environments.
A webinar that is happening on Tuesday, June 23 entitled "Addressing the Unique IT Security Risks Posed by the Virtual Data Center". You can use the same site to watch a replay after June 23rd.

In this blog post I am going to first review what are some of the identity management challenges in VMware environments, discuss what VMware provides out-of-the-box in terms of Active Directory integration and its limitations, and discuss what Centrify uniquely offers. Many thanks to David McNeely, our Director of Product Management, for providing me the lion share of this content.

What Are the Account Management Challenges in VMware ESX?

To set up and manage each of the virtual systems on an ESX host machine, an administrator needs to log in to one of the VMware administrative interfaces. Since the ESX Server runs in effect a version of Linux, the standard method for logging in to the host system via the Service Console is very similar to logging in to a Linux system: there is a root user, and additional users and groups can be configured and stored on the local host system using the same /etc/passwd and /etc/group method that standard Linux uses. Administrators with the appropriate set of privileges, called "roles" in VMware Infrastructure, can create or delete virtual machines, control various functions associated with each machine, dynamically provision and manage the computing capacity available to each machine, as well as monitor individual machine's performance.

Additionally, to perform system-level operations, an administrator needs root-level privileges within the Linux kernel operating environment in order to carry out several operational commands via the Service Console. VMware provides other administrative interfaces, including the Virtual Infrastructure Client, the Web Management User Interface, and the VMware Infrastructure Management Agent; all these interfaces require the user to log in with a credential that is recognized by the ESX host and authorized to perform the actions being requested.

Image: VMware management interfaces

Although ESX by default uses a local store of users and passwords for authentication, it is also possible to use other methods to validate user logins since its authentication framework is PAM (Pluggable Authentication Modules). PAM can be configured to support other authentication mechanisms and use a central directory service for authentication and user information storage.

Centralized directory services offer numerous benefits to the administrator, including:

User accounts can be stored in a single, secure database available to many different systems as opposed to being stored and managed on each system.
Managing permissions and policies can be centralized, resulting in better security for each system.
Password management can be centralized and consistent user names applied.
Provisioning and de-provisioning user accounts can be done very quickly from a single administrative system.

Since most enterprise organizations use Active Directory, have existing processes, and have trained staff for the administration of accounts and security policies, Centrify has developed an identity and access management solution, the Centrify Suite, to integrate non-Windows systems into Active Directory. Centrify Suite provides an agent which enables ESX systems to leverage Active Directory for centralized directory services, authentication, role-based privilege management, and policy controls.

Image: Active Directory-integrated login with the Centrify Suite.

But aha you say, doesn't VMware provide Active Directory integration? Lets take a look at what they offer.

Comparing Centrify for Active Directory Integration with VMware Native Active Directory Integration

VMware published a technical note titled Enabling Active Directory Authentication with ESX Server (http://www.vmware.com/pdf/esx3_esxcfg_auth_tn.pdf). This paper discusses using the esxcfg-auth tool to set up Kerberos authentication through Active Directory. The command syntax of this tool is as follows:


esxcfg-auth --enabled -addomain=<domain name> --addc=<domain controller name>

This tool configures PAM and modifies the ESX server configuration to do login authentication from the specified Active Directory domain controller. After executing the preceding command, you then create a local account for each user who requires access to the ESX server, making sure that the user ID is exactly the same as his Active Directory user name.

This process would then need to be repeated for every ESX server in your environment. While these steps do enable authentication from an Active Directory system for an ESX Server, it does not leverage Active Directory for authorization, centralized directory services or policy management. Specifically, the methods outlined in this paper have the following serious shortcomings:

This is not a truly integrated solution as it does not offer a single source for defining, managing and authenticating user accounts. While the esxcfg-auth tool allows you to use Active Directory to authenticate users, you cannot use Active Directory to define and manage user accounts for ESX. User accounts are still created and maintained on each ESX server.
The process to enable Active Directory authentication for every user who requires access to the ESX server is clumsy. For each individual user, you must also create a corresponding user account on the ESX host server. Authorized users can log in under two scenarios: (a) if they have a valid Active Directory password associated with the user name they provided and if they have a local account in /etc/passwd that also matches this user name, or (b) if they have a local user name and password on the system. This means that the administrator must manually synchronize the user account information between authorized Active Directory users and each ESX server, and carefully map intended user access to actual possibilities for user access.
If the network goes down or the Active Directory system is unavailable, users who use Active Directory for authentication will not be able log in to the ESX server. Credentials are not cached, and there is no provision for the underlying Kerberos authentication session to fail over to a backup system.
Given the issues with the previous point, the paper recommends not using Active Directory authentication for the root account. This means that there are few controls over who has access to the superuser account on each ESX server and also means that the root user password needs to be set manually for every ESX server.
There is also more network traffic with each Kerberos transaction since this method does not support any type of caching.
The machine name for the Active Directory / Kerberos server is hard-coded in the system files for each ESX server. If the name of the closest domain controller changes, the administrator needs to manually update this information in each system file on each ESX server.
The ESX server is not joined to the domain, so Active Directory has no knowledge of the system or any control over the ESX server. This means that if the administrator wanted to temporarily restrict access to an ESX server or a whole set of ESX servers, he or she would have no way to accomplish this from Active Directory.
The paper does not provide guidance on how to set up FTP or SSH for accessing the ESX server. Typically, having access to these services is essential for system administrators. Also, there is no guidance on setting up this new authentication method for all management session types (Remote Console, VMware Management Interface, etc.).
The paper acknowledges that this method for authentication will fail if the user is a member of more than 15 Active Directory groups, which in a large enterprise is quite common.
There is no guidance on how to track access to the ESX server using this implementation.

Given all of these challenges, the proposed solution in the VMware paper will be untenable for many organizations. VMware offers another product, VirtualCenter, which provides centralized administration and management for ESX servers connected on a network. It acts as a control node for configuring, provisioning and managing a virtualized IT environment consisting of ESX servers. For a VI Client that is connected to a VirtualCenter server, authentication and authorization are performed via an Active Directory service. Authorized VirtualCenter users are selected from the Windows domain list referenced in VirtualCenter or are local Windows users on the VirtualCenter host. Similarly, VirtualCenter groups are derived from Active Directory in the connected Windows domain. Both Active Directory-based users and groups are then granted permissions ("roles") within VirtualCenter. However, on the back end, VirtualCenter still uses the standard Linux authentication mechanism. Whenever an ESX server host is added to it, VirtualCenter creates a Linux user account (vpxuser) that has root privileges. This account is used only to authenticate the connection between the host and VirtualCenter.

Although VirtualCenter resolves the issue of separate password management and account management in the esxcfg-auth tool, it has a number of shortcomings in its integration with Active Directory:

VirtualCenter serves as a central point to manage multiple virtual machines and resources that are distributed over many ESX server hosts. Therefore, it is not cost-effective for small deployments.
This is still not a seamlessly integrated solution. You cannot use VirtualCenter to manually create and remove ESX users or groups, or to view and modify their properties such as passwords. You will have to use the Microsoft tools for user account and password management.
There are still occasions when you need to access an ESX server host via other mechanisms; for example, when VirtualCenter is unavailable or has lost its connection to the domain controller. In addition, there are still a few administrative tasks that must be performed directly on the ESX host and not through VirtualCenter.

Can Centrify DirectControl provide a better integration with Active Directory? Yes it can, as described next.

Addressing the Authentication Challenges with Centrify DirectControl

Centrify DirectControl is engineered not only to be easy to use but also to be a completely integrated authentication, authorization, directory and policy solution. As a result, the issues highlighted in the previous section are fully resolved with DirectControl. Specifically:

Unlike the esxcfg-auth tool, DirectControl provides unified account and password management. There is no need to create a local user and map it to the Active Directory account for every user that you want to grant access to the ESX Server host.
The DirectControl integration with Active Directory is seamless from a user interface perspective. You cannot create or manage Active Directory users and groups via VirtualCenter, but Centrify extends the native ADUC MMC with UNIX properties for user, group and computer objects, which enables you to use the same tool to manage not only ESX users and groups but also the Active Directory account information associated with them. In addition, Centrify provides the DirectControl Administrator Console so you can view and modify all the attributes of Active Directory's user, group and computer objects, including the DirectControl ones.
With the Centrify solution, authorization is handled from one central place using the DirectControl Administrator Console. The administrator has the ability to create an explicit access list of users for each ESX server. Through the use of Centrify Zones, ESX administrators can be members of their own Zone of ESX servers, further simplifying the access control for those systems. In addition, users can be further restricted based on policies such as authorized access times. Authorized users can also be placed in Active Directory groups that are visible from ESX as though they were local groups. This allows a high level of fine-grained access control for each ESX server. If changes need to be made, they can be done from a single point of administration, the DirectControl Administrator Console.

Image: Zone-based user access controls

DirectControl fully supports the caching of login credentials. If a user has logged in to the ESX server at least once, then he or she can continue to log in to that system even if the network is down. Or, the administrator can configure users or groups for pre-validation so that they can access offline machines using their Active Directory credentials without having logged in previously. Also, when a user logs in for the first time, DirectControl automatically creates a home directory environment for the user if one does not already exist. DirectControl can also automatically find the closest available Active Directory domain controller, so that if one domain controller is taken offline, another can be automatically used without the need to reconfigure the ESX server.
Since login credentials are cached, network traffic is reduced. This is an important consideration where multiple virtual machines are sharing the same network interface with the host ESX server.
Login credentials can also be pre-cached for those administrators who must always be able to log in with their account regardless of the state of the network connectivity, such as at a remote location with a down WAN link where the ESX system requires administrative access for maintenance.
DirectControl includes a feature for root user mapping. This means the root account for every machine can be mapped to an Active Directory user, and password control is maintained in a central place. With support for offline caching, the root user can still log in to ESX server even if the Active Directory system is unavailable.
As mentioned in a previous point, DirectControl manages the interactions with the Active Directory domain controller and automatically finds the closest controller for each controller request.
With DirectControl, the ESX server is joined to the Active Directory domain. As with other systems in the domain, the administrator has full control over access to the ESX server, including temporarily disabling logins — for example, during a maintenance period.
DirectControl automatically configures access to popular services such as FTP, Telnet and SSH to use secured authentication via Kerberos to Active Directory. For example, Centrify provides a compiled version of the latest OpenSSH distribution that is linked with the DirectControl Kerberos libraries to automatically support PAM and Kerberos for single sign-on access.
DirectControl ensures that a single authentication method is used across all supported VMware management session types, including the local Service Console, VMware Management Interface (VI Client and VI Web Access) as well as Remote Console sessions such as via the SSH protocol.
DirectControl does not impose any limits on group membership.
DirectControl's integration with Active Directory has proven to work in complex environments — for example, in a topology with multiple forests that requires one- or two-way trusts.

In addition, Centrify DirectControl has other advantages beyond providing identity management:

DirectControl fully supports Microsoft Group Policy and includes an extensive set of policies out-of-the-box for security and configuration management. You can use DirectControl's built-in Group Policy engine to distribute computer and user policies to a set of ESX servers. Such policies can copy configuration files to target systems, manage various configuration parameters such as login settings, password prompts, password caching and Kerberos settings, as well as define sudo permissions. For added flexibility, you can even create your own custom policies specifically tailored for your virtualized IT infrastructure. Through the deployment of policies to your ESX servers, you ensure consistent machine configuration and further control the ESX session behavior. As a result you streamline your IT operations and reduce administrative costs.
In addition, since ESX administration can be performed through a remote connection via the SSH protocol, you can also use the Centrify SSH Group Policies to configure who can connect to the host using SSH, such as only users of a specific group or to prevent root login via SSH.
DirectControl is supported on most of the UNIX and Linux platforms available today, plus Mac OS X, so customers can have a consistent Active Directory integration solution across their non-Microsoft platforms.
This integration can also be extended to the Linux and UNIX virtual machines running inside ESX server. Each virtual machine, or groups of machines, can be managed within a dedicated Zone. This is particularly useful when ESX server is used for outsourcing environments where identity groups from different organizations need to be managed individually and isolated from each other.
The DirectControl identity management solution extends beyond validating login sessions. DirectControl can also support applications that take advantage of LDAP, Kerberos, GSSAPI or SPNEGO APIs for directory services and authentication. This means customers could design custom applications for ESX (such as a customer bill.back system for virtual machine usage) based on validated identities stored in Active Directory.
Centrify DirectAuthorize provides an alternative method of controlling user privileges by leveraging Active Directory to centrally manage and enforce role-based entitlements. DirectAuthorize provides fine-grained control over user access and privileges on UNIX and Linux systems, including ESX. By controlling which methods users access systems and what they can do once logged in, DirectAuthorize enables organizations to lock down sensitive systems and eliminate uncontrolled use of root accounts and passwords.
ESX servers are typically one of the most crucial components in a virtualized infrastructure, and hence should be protected from security intrusion in the IT environment. Thus, all administrative access and activities on an ESX server should be logged and tracked. Centrify DirectAudit complements DirectControl by providing detailed and non-intrusive recording of UNIX and Linux user sessions, which gives auditors and security officers ad-hoc search and reporting capabilities. By using DirectAudit, the auditor now has an audit trail of which users accessed what systems, what commands they executed, and what changes they made to key files and data. To limit the amount of output, he can further restrict the session auditing to a specific user or a specific shell.