Nova如何统计节点硬件资源

摘要：引言当我们在使用那些建设在之上的云平台服务的时候，往往在概览页面都有一个明显的位置用来展示当前集群的一些资源使用情况，如，，内存，硬盘等资源的总量使用量剩余量。如上，就是统计节点硬件资源的整个逻辑过程为例。

当我们在使用那些建设在OpenStack之上的云平台服务的时候，往往在概览页面都有一个明显的位置用来展示当前集群的一些资源使用情况，如，CPU，内存，硬盘等资源的总量、使用量、剩余量。而且，每当我们拓展集群规模之后，概览页面上的资源总量也会自动增加，我们都熟知，OpenStack中的Nova服务负责管理这些计算资源，那么你有没有想过，它们是如何被Nova服务获取的吗？

我们知道，统计资源的操作属于Nova服务内部的机制，考虑到资源统计结果对后续操作(如创建虚拟机，创建硬盘)的重要性，我们推断该机制的运行顺序一定先于其他服务。

通过上述简单的分析，再加上一些必要的Debug操作，我们得出：
该机制的触发点位于nova.service.WSGIService.start方法中：

    def start(self):
        """Start serving this service using loaded configuration.

        Also, retrieve updated port number in case "0" was passed in, which
        indicates a random port should be used.

        :returns: None

        """
        if self.manager:
            self.manager.init_host()
            self.manager.pre_start_hook()
            if self.backdoor_port is not None:
                self.manager.backdoor_port = self.backdoor_port
        self.server.start()
        if self.manager:
            self.manager.post_start_hook()

其中，self.manager.pre_start_hook()的作用就是去获取资源信息,它的直接调用为nova.compute.manager.pre_start_hook如下：

    def pre_start_hook(self):
        """After the service is initialized, but before we fully bring
        the service up by listening on RPC queues, make sure to update
        our available resources (and indirectly our available nodes).
        """
        self.update_available_resource(nova.context.get_admin_context())
...
    @periodic_task.periodic_task
    def update_available_resource(self, context):
        """See driver.get_available_resource()

        Periodic process that keeps that the compute host"s understanding of
        resource availability and usage in sync with the underlying hypervisor.

        :param context: security context
        """
        new_resource_tracker_dict = {}
        nodenames = set(self.driver.get_available_nodes())
        for nodename in nodenames:
            rt = self._get_resource_tracker(nodename)
            rt.update_available_resource(context)
            new_resource_tracker_dict[nodename] = rt

        # Delete orphan compute node not reported by driver but still in db
        compute_nodes_in_db = self._get_compute_nodes_in_db(context,
                                                            use_slave=True)

        for cn in compute_nodes_in_db:
            if cn.hypervisor_hostname not in nodenames:
                LOG.audit(_("Deleting orphan compute node %s") % cn.id)
                cn.destroy()

        self._resource_tracker_dict = new_resource_tracker_dict

上述代码中的rt.update_available_resource()的直接调用实为nova.compute.resource_tracker.update_available_resource()如下:

    def update_available_resource(self, context):
        """Override in-memory calculations of compute node resource usage based
        on data audited from the hypervisor layer.

        Add in resource claims in progress to account for operations that have
        declared a need for resources, but not necessarily retrieved them from
        the hypervisor layer yet.
        """
        LOG.audit(_("Auditing locally available compute resources"))
        resources = self.driver.get_available_resource(self.nodename)

        if not resources:
            # The virt driver does not support this function
            LOG.audit(_("Virt driver does not support "
                 ""get_available_resource"  Compute tracking is disabled."))
            self.compute_node = None
            return
        resources["host_ip"] = CONF.my_ip

        # TODO(berrange): remove this once all virt drivers are updated
        # to report topology
        if "numa_topology" not in resources:
            resources["numa_topology"] = None

        self._verify_resources(resources)
        
        self._report_hypervisor_resource_view(resources)

        return self._update_available_resource(context, resources)

上述代码中的self._update_available_resource的作用是根据计算节点上的资源实际使用结果来同步数据库记录，这里我们不做展开；self.driver.get_available_resource()的作用就是获取节点硬件资源信息，它的实际调用为：

class LibvirtDriver(driver.ComputeDriver):
    def get_available_resource(self, nodename):
        """Retrieve resource information.

        This method is called when nova-compute launches, and
        as part of a periodic task that records the results in the DB.

        :param nodename: will be put in PCI device
        :returns: dictionary containing resource info
        """

        # Temporary: convert supported_instances into a string, while keeping
        # the RPC version as JSON. Can be changed when RPC broadcast is removed
        stats = self.get_host_stats(refresh=True)
        stats["supported_instances"] = jsonutils.dumps(
                stats["supported_instances"])
        return stats
        
    def get_host_stats(self, refresh=False):
        """Return the current state of the host.

        If "refresh" is True, run update the stats first.
        """
        return self.host_state.get_host_stats(refresh=refresh)
        
        def _get_vcpu_total(self):
        """Get available vcpu number of physical computer.

        :returns: the number of cpu core instances can be used.

        """
        if self._vcpu_total != 0:
            return self._vcpu_total

        try:
            total_pcpus = self._conn.getInfo()[2] + 1
        except libvirt.libvirtError:
            LOG.warn(_LW("Cannot get the number of cpu, because this "
                         "function is not implemented for this platform. "))
            return 0

        if CONF.vcpu_pin_set is None:
            self._vcpu_total = total_pcpus
            return self._vcpu_total

        available_ids = hardware.get_vcpu_pin_set()
        if sorted(available_ids)[-1] >= total_pcpus:
            raise exception.Invalid(_("Invalid vcpu_pin_set config, "
                                      "out of hypervisor cpu range."))
        self._vcpu_total = len(available_ids)
        return self._vcpu_total

.....
class HostState(object):
    """Manages information about the compute node through libvirt."""
    def __init__(self, driver):
        super(HostState, self).__init__()
        self._stats = {}
        self.driver = driver
        self.update_status()

    def get_host_stats(self, refresh=False):
        """Return the current state of the host.

        If "refresh" is True, run update the stats first.
        """
        if refresh or not self._stats:
            self.update_status()
        return self._stats
        
    def update_status(self):
        """Retrieve status info from libvirt."""
        ...
        data["vcpus"] = self.driver._get_vcpu_total()
        data["memory_mb"] = self.driver._get_memory_mb_total()
        data["local_gb"] = disk_info_dict["total"]
        data["vcpus_used"] = self.driver._get_vcpu_used()
        data["memory_mb_used"] = self.driver._get_memory_mb_used()
        data["local_gb_used"] = disk_info_dict["used"]
        data["hypervisor_type"] = self.driver._get_hypervisor_type()
        data["hypervisor_version"] = self.driver._get_hypervisor_version()
        data["hypervisor_hostname"] = self.driver._get_hypervisor_hostname()
        data["cpu_info"] = self.driver._get_cpu_info()
        data["disk_available_least"] = _get_disk_available_least()
        ...

注意get_available_resource方法的注释信息，完全符合我们开始的推断。我们下面单以vcpus为例继续调查资源统计流程，self.driver._get_vcpu_total的实际调用为LibvirtDriver._get_vcpu_total(上述代码中已给出)，如果配置项vcpu_pin_set没有生效，那么得到的_vcpu_total的值为self._conn.getInfo()[2]（self._conn可以理解为libvirt的适配器，它代表与kvm,qemu等底层虚拟化工具的抽象连接，getInfo()就是对libvirtmod.virNodeGetInfo的一次简单的封装，它的返回值是一组数组，其中第三个元素就是vcpus的数量），我们看到这里基本就可以了，再往下就是libvirt的C语言代码而不是Python的范畴了。

另一方面，如果我们配置了vcpu_pin_set配置项，那么该配置项就被hardware.get_vcpu_pin_set方法解析成一个可用CPU位置索引的集合，再通过对该集合求长后，我们也能得到最终想要的vcpus的数量。

如上，就是Nova统计节点硬件资源的整个逻辑过程(vcpus为例)。