Ksplice

Fixing Security Vulnerabilities in Linux

Security vulnerabilities are some of the hardest bugs to discover yet they can have the largest impact. At Ksplice, we spend a lot of time looking at security vulnerabilities and seeing how they are fixed. We use automated tools such as the Trinity syscall fuzzer and the Kernel Address Sanitizer (KASan) to aid our process. In this blog post we’ll go over some case studies of recent vulnerabilities and show you how you can avoid them in your code.

CVE-2013-7339 and CVE-2014-2678

These two are very similar NULL pointer dereferences when trying to bind an RDS socket without having an RDS device. This is an oversight that happens quite often in hardware-specific code in the kernel. It is easy for developers to assume that a piece of hardware always exists since all their dev machines have it, but that sometimes leads to other possible hardware configurations left untested. In this example the code makes a seemingly reasonable assumption that using RDS sockets without RDS hardware doesn’t really make sense.

The issue is pretty simple as we can see from this fix:

diff --git a/net/rds/ib.c b/net/rds/ib.c
index b4c8b00..ba2dffe 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -338,7 +338,8 @@ static int rds_ib_laddr_check(__be32 addr)
   ret = rdma_bind_addr(cm_id, (struct sockaddr *)&sin);
   /* due to this, we will claim to support iWARP devices unless we
      check node_type. */
-     if (ret || cm_id->device->node_type != RDMA_NODE_IB_CA)
+     if (ret || !cm_id->device ||
+         cm_id->device->node_type != RDMA_NODE_IB_CA)
                                   ret = -EADDRNOTAVAIL;

                                   rdsdebug("addr %pI4 ret %d node type %d\n",

Generally we are allowed to bind an address without a physical device so we can reach this code without any RDS hardware. Sadly, this code wrongly assumes that a devices exists at this point and that cm_id->device is not NULL leading to a NULL pointer dereference.

These type of issues are usually caught early in -next as that exposes the code to various users and hardware configurations, but this one managed to slip through somehow.

There are many variations of the scenario where the hardware specific and other kernel code doesn’t handle
cases which “don’t make sense”. Another recent example is dlmfs. The kernel would panic when trying to create a directory on it – something that doesn’t happen in regular usage of dlmfs.

CVE-2014-3940

This one is interesting and very difficult to stumble upon by accident. It’s a race condition that is only possible during the migration of huge pages between NUMA nodes, so the window of opportunity is *very* small. It can be triggered by trying to dump the NUMA maps of a process while its memory is being moved around. What happens is that the code trying to dump memory makes invalid memory accesses because it does not check the presence of the memory beforehand.

When we dump NUMA maps we need to walk memory entries using walk_page_range():


        /*
         * Handle hugetlb vma individually because pagetable
         * walk for the hugetlb page is dependent on the
         * architecture and we can't handled it in the same
         * manner as non-huge pages.
         */
        if (walk->hugetlb_entry && (vma->vm_start <= addr) &&
            is_vm_hugetlb_page(vma)) {
                if (vma->vm_end < next)
                        next = vma->vm_end;
                /*
                 * Hugepage is very tightly coupled with vma,
                 * so walk through hugetlb entries within a
                 * given vma.
                 */
                err = walk_hugetlb_range(vma, addr, next, walk);
                if (err)
                        break;
                pgd = pgd_offset(walk->mm, next);
                continue;
        }

When walk_page_range() detects a hugepage it calls walk_hugetlb_range(), which calls the proc’s callback (provided by walk->hugetlb_entry()) for each page individually:


        pte_t *pte;
        int err = 0;

        do {
                next = hugetlb_entry_end(h, addr, end);
                pte = huge_pte_offset(walk->mm, addr & hmask);
                if (pte && walk->hugetlb_entry)
                        err = walk->hugetlb_entry(pte, hmask, addr, next, walk);
                if (err)
                        return err;
        } while (addr = next, addr != end);

Note that the callback is executed for each pte; even for those that are not present in memory (pte_present(*pte) would return false in that case). This is done by the walker code because it was assumed that callback functions might want to handle that scenario for some reason. In the past there was no way for a huge pte to be absent, but that changed when the hugepage migration was introduced. During page migration unmap_and_move_huge_page() removes huge ptes:


if (page_mapped(hpage)) {

try_to_unmap(hpage,
TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);

page_was_mapped = 1;

}

Unfortunately, some callbacks were not changed to deal with this new possibility. A notable example is gather_pte_stats(), which tries to lock a non-existent pte:


        orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);

This can cause a panic if it happens during a tiny window inside unmap_and_move_huge_page().

Dumping NUMA maps doesn’t happen too often and is mainly used for testing/debugging, so this bug has lived there for quite a while and was made visible only recently when hugepage migration was added.

It’s also common that adding userspace interfaces to trigger kernel code which doesn’t get called often exposes many issues. This happened recently when the firmware loading code was exposed to userspace.

CVE-2014-4171

This one also falls into the category of “doesn’t make sense” because it involves repeated page faulting of memory that we marked as unwanted. When this happens shmem tries to remove a block of memory, but since it’s getting faulted over and over again shmem will hang waiting until it’s available for removal. Meanwhile other filesystem operations will be blocked, which is bad because that memory may never become available for removal.

When we’re faulting a shmem page in, shmem_fault() would grab a reference to the page:


static int shmem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
[...]
error = shmem_getpage(inode, vmf->pgoff, &vmf->page, SGP_CACHE, &ret);

But because shmem_fallocate() holds i_mutex this means that shmem_fallocate() can wait forever until it can free up that page. This, in turn means that that filesystem is stuck waiting for shmem_fallocate() to complete.

Beyond that, punching holes in files and marking memory as unwanted are not common operations; especially not on a shmem filesystem. This means that those code paths are very untested.

CVE-2014-4943

This is a privilege escalation which was found using KASan. We’ve noticed that as a result of a call to a PPPOL2TP ioctl an uninitialized address inside a struct was being read. Further investigation showed that this is the result of a confusion about the type of the struct that was being accessed.

When we call setsockopt from userspace on a PPPOL2TP socket in userspace we’ll end up in pppol2tp_setsockopt() which will look at the level parameter to see if the sockopt operation is intended for PPPOL2TP or the underlying UDP socket:


   if (level != SOL_PPPOL2TP)
      return udp_prot.setsockopt(sk, level, optname, optval, optlen);

PPPOL2TP tries to be helpful here and allows userspace to set UDP sockopts rather than just PPPOL2TP ones. The problem here is that UDP‘s setsockopt expects a udp_sock:


 int udp_lib_setsockopt(struct sock *sk, int level, int optname,
                        char __user *optval, unsigned int optlen,
                        int (*push_pending_frames)(struct sock *))
 {
         struct udp_sock *up = udp_sk(sk);

But instead it’s getting just a sock struct.

It’s possible to leverage this struct confusion to achieve privilege escalation. We can overwrite the function pointer in the struct to point to code of our choice. Then we can trigger the execution of this code by making another socket operation. The piece of code that allowed for this vulnerability was added for convenience, but no one ever needed it, and it was never tested.

Conclusion

We hope that this exposition of straightforward and more subtle kernel bugs will remind of the importance of looking at code from a new perspective and encourage the developer community to contribute to and create new tools and methodologies for detecting and preventing bugs in the kernel.

Fixing Security Vulnerabilities in Linux

Security vulnerabilities are some of the hardest bugs to discover yet they can have the largest impact. At Ksplice, we spend a lot of time looking at security vulnerabilities and seeing how they are fixed. We use automated tools such as the Trinity syscall fuzzer and the Kernel Address Sanitizer (KASan) to aid our process. In this blog post we’ll go over some case studies of recent vulnerabilities and show you how you can avoid them in your code.
CVE-2013-7339 and CVE-2014-2678

These two are very similar NULL pointer dereferences when trying to bind an RDS socket without having an RDS device. This is an oversight that happens quite often in hardware-specific code in the kernel. It is easy for developers to assume that a piece of hardware always exists since all their dev machines have it, but that sometimes leads to other possible hardware configurations left untested. In this example the code makes a seemingly reasonable assumption that using RDS sockets without RDS hardware doesn’t really make sense.

The issue is pretty simple as we can see from this fix:

diff –git a/net/rds/ib.c b/net/rds/ib.cindex b4c8b00..ba2dffe 100644— a/net/rds/ib.c+++ b/net/rds/ib.c@@ -338,7 +338,8 @@ static int rds_ib_laddr_check(__be32 addr) ret = rdma_bind_addr(cm_id, (struct sockaddr *)&sin); /* due to this, we will claim to support iWARP devices unless we check node_type. */- if (ret || cm_id->device->node_type != RDMA_NODE_IB_CA)+ if (ret || !cm_id->device ||+ cm_id->device->node_type != RDMA_NODE_IB_CA) ret = -EADDRNOTAVAIL; rdsdebug(“addr %pI4 ret %d node type %d\n”,

Generally we are allowed to bind an address without a physical device so we can reach this code without any RDS hardware. Sadly, this code wrongly assumes that a devices exists at this point and that cm_id->device is not NULL leading to a NULL pointer dereference.

These type of issues are usually caught early in -next as that exposes the code to various users and hardware configurations, but this one managed to slip through somehow.

There are many variations of the scenario where the hardware specific and other kernel code doesn’t handle cases which “don’t make sense”. Another recent example is dlmfs. The kernel would panic when trying to create a directory on it – something that doesn’t happen in regular usage of dlmfs.

CVE-2014-3940

This one is interesting and very difficult to stumble upon by accident. It’s a race condition that is only possible during the migration of huge pages between NUMA nodes, so the window of opportunity is *very* small. It can be triggered by trying to dump the NUMA maps of a process while its memory is being moved around. What happens is that the code trying to dump memory makes invalid memory accesses because it does not check the presence of the memory beforehand.

When we dump NUMA maps we need to walk memory entries using walk_page_range():

/* * Handle hugetlb vma individually because pagetable * walk for the hugetlb page is dependent on the * architecture and we can’t handled it in the same * manner as non-huge pages. */ if (walk->hugetlb_entry && (vma->vm_start <= addr) && is_vm_hugetlb_page(vma)) { if (vma->vm_end < next) next = vma->vm_end; /* * Hugepage is very tightly coupled with vma, * so walk through hugetlb entries within a * given vma. */ err = walk_hugetlb_range(vma, addr, next, walk); if (err) break; pgd = pgd_offset(walk->mm, next); continue; }

When walk_page_range() detects a hugepage it calls walk_hugetlb_range(), which calls the proc’s callback (provided by walk->hugetlb_entry()) for each page individually:

pte_t *pte; int err = 0; do { next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask); if (pte && walk->hugetlb_entry) err = walk->hugetlb_entry(pte, hmask, addr, next, walk); if (err) return err; } while (addr = next, addr != end);

Note that the callback is executed for each pte; even for those that are not present in memory (pte_present(*pte) would return false in that case). This is done by the walker code because it was assumed that callback functions might want to handle that scenario for some reason. In the past there was no way for a huge pte to be absent, but that changed when the hugepage migration was introduced. During page migration unmap_and_move_huge_page() removes huge ptes:

if (page_mapped(hpage)) {

try_to_unmap(hpage, TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);

page_was_mapped = 1;

}

Unfortunately, some callbacks were not changed to deal with this new possibility. A notable example is gather_pte_stats(), which tries to lock a non-existent pte:

orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);

This can cause a panic if it happens during a tiny window inside unmap_and_move_huge_page().

Dumping NUMA maps doesn’t happen too often and is mainly used for testing/debugging, so this bug has lived there for quite a while and was made visible only recently when hugepage migration was added.

It’s also common that adding userspace interfaces to trigger kernel code which doesn’t get called often exposes many issues. This happened recently when the firmware loading code was exposed to userspace.

CVE-2014-4171

This one also falls into the category of “doesn’t make sense” because it involves repeated page faulting of memory that we marked as unwanted. When this happens shmem tries to remove a block of memory, but since it’s getting faulted over and over again shmem will hang waiting until it’s available for removal. Meanwhile other filesystem operations will be blocked, which is bad because that memory may never become available for removal.

When we’re faulting a shmem page in, shmem_fault() would grab a reference to the page:

static int shmem_fault(struct vm_area_struct *vma, struct vm_fault *vmf) { […] error = shmem_getpage(inode, vmf->pgoff, &vmf->page, SGP_CACHE, &ret);

But because shmem_fallocate() holds i_mutex this means that shmem_fallocate() can wait forever until it can free up that page. This, in turn means that that filesystem is stuck waiting for shmem_fallocate() to complete.

Beyond that, punching holes in files and marking memory as unwanted are not common operations; especially not on a shmem filesystem. This means that those code paths are very untested.

CVE-2014-4943

This is a privilege escalation which was found using KASan. We’ve noticed that as a result of a call to a PPPOL2TP ioctl an uninitialized address inside a struct was being read. Further investigation showed that this is the result of a confusion about the type of the struct that was being accessed.

When we call setsockopt from userspace on a PPPOL2TP socket in userspace we’ll end up in pppol2tp_setsockopt() which will look at the level parameter to see if the sockopt operation is intended for PPPOL2TP or the underlying UDP socket:

if (level != SOL_PPPOL2TP) return udp_prot.setsockopt(sk, level, optname, optval, optlen);

PPPOL2TP tries to be helpful here and allows userspace to set UDP sockopts rather than just PPPOL2TP ones. The problem here is that UDPs setsockopt expects a udp_sock:

int udp_lib_setsockopt(struct sock *sk, int level, int optname, char __user *optval, unsigned int optlen, int (*push_pending_frames)(struct sock *)) { struct udp_sock *up = udp_sk(sk);

But instead it’s getting just a sock struct.

It’s possible to leverage this struct confusion to achieve privilege escalation. We can overwrite the function pointer in the struct to point to code of our choice. Then we can trigger the execution of this code by making another socket operation. The piece of code that allowed for this vulnerability was added for convenience, but no one ever needed it, and it was never tested.

Conclusion

We hope that this exposition of straightforward and more subtle kernel bugs will remind of the importance of looking at code from a new perspective and encourage the developer community to contribute to and create new tools and methodologies for detecting and preventing bugs in the kernel.

LKML: Live patching for 3.20

https://lkml.org/lkml/2015/2/9/534
Building on the original kSplice idea and combining the efforts of the work done at Red Hat and SuSE, common infrastructure is now ready to be put into the Linux 3.20 mainline kernel – Red Hat and SuSE have already committed to using this.
I still reckon it’s freaky trickery, but heck – it works, and it’s great for server environments that have no redundancy (I prefer to fix that issue!) and can’t afford any downtime.

Ksplice SNMP Plugin

The Ksplice team is happy to announce the release of an SNMP plugin for Ksplice, available today on the Unbreakable Linux Network. The plugin will let you use Oracle Enterprise Manager to monitor the status of Ksplice on all of your systems, but it will also work with any monitoring solution that is SNMP compatible.

Installation

You’ll find the plugin on Ksplice channel for your distribution and architecture. For Oracle Linux 6 on x86_64 that’s ol6_x86_64_ksplice. Install the plugin by running (as root):

yum install ksplice-snmp-plugin Configuration

If you haven’t set up SNMP before, you’ll need to do a little bit of configuration. Included below is a sample /etc/snmp/snmpd.conf file to try out the plugin:

# Setting up permissions# ======================com2sec local localhost publiccom2sec mynet 10.10.10.0/24 publicgroup local v1 localgroup local v2c localgroup local usm localgroup mynet v1 mynetgroup mynet v2c mynetgroup mynet usm mynetview all included .1 80access mynet “” any noauth exact all none noneaccess local “” any noauth exact all all nonesyslocation Oracle Linux 6syscontact sysadmin <root@localhost># Load the plugin# ===============dlmod kspliceUptrack /usr/lib64/ksplice-snmp/kspliceUptrack.so

You’ll want to replace the IP address next to com2sec with the address of your local network, or the address of your SNMP monitoring software. If you are running on a 32 bit architecture x86), replace lib64 in the dlmod path with lib. The above configuration is an example, intended only for testing purposes. For more information about configuring SNMP, check out the SNMP documentation, including the man pages for snmpd and snmpd.conf.

Examples

You can test out your configuration by using the snmpwalk command and verifying the responses. Some examples:

Displaying the installed version of Ksplice:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceVersionKSPLICE-UPTRACK-MIB::kspliceVersion.0 = STRING: 1.2.12

Checking if a kernel has all available updates installed:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatusKSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate

Displaying and comparing the kernel installed on disk with the Ksplice effective version:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceBaseKernelKSPLICE-UPTRACK-MIB::kspliceBaseKernel.0 = STRING: 2.6.18-274.3.1.el5$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceEffectiveKernelKSPLICE-UPTRACK-MIB::kspliceEffectiveKernel.0 = STRING: 2.6.18-274.3.1.el5

Displaying a list of all installed updates:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTable

In this case, there are none. This is why the base kernel version and effective kernel version are the same, and why this kernel is out of date.

Displaying a list of updates that can be installed right now, including their description:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTableKSPLICE-UPTRACK-MIB::kspliceavailIndex.0 = INTEGER: 0KSPLICE-UPTRACK-MIB::kspliceavailIndex.1 = INTEGER: 1KSPLICE-UPTRACK-MIB::kspliceavailIndex.2 = INTEGER: 2KSPLICE-UPTRACK-MIB::kspliceavailIndex.3 = INTEGER: 3KSPLICE-UPTRACK-MIB::kspliceavailIndex.4 = INTEGER: 4KSPLICE-UPTRACK-MIB::kspliceavailIndex.5 = INTEGER: 5KSPLICE-UPTRACK-MIB::kspliceavailIndex.6 = INTEGER: 6KSPLICE-UPTRACK-MIB::kspliceavailIndex.7 = INTEGER: 7KSPLICE-UPTRACK-MIB::kspliceavailIndex.8 = INTEGER: 8KSPLICE-UPTRACK-MIB::kspliceavailIndex.9 = INTEGER: 9KSPLICE-UPTRACK-MIB::kspliceavailIndex.10 = INTEGER: 10KSPLICE-UPTRACK-MIB::kspliceavailIndex.11 = INTEGER: 11KSPLICE-UPTRACK-MIB::kspliceavailIndex.12 = INTEGER: 12KSPLICE-UPTRACK-MIB::kspliceavailIndex.13 = INTEGER: 13KSPLICE-UPTRACK-MIB::kspliceavailIndex.14 = INTEGER: 14KSPLICE-UPTRACK-MIB::kspliceavailIndex.15 = INTEGER: 15KSPLICE-UPTRACK-MIB::kspliceavailIndex.16 = INTEGER: 16KSPLICE-UPTRACK-MIB::kspliceavailIndex.17 = INTEGER: 17KSPLICE-UPTRACK-MIB::kspliceavailIndex.18 = INTEGER: 18KSPLICE-UPTRACK-MIB::kspliceavailIndex.19 = INTEGER: 19KSPLICE-UPTRACK-MIB::kspliceavailIndex.20 = INTEGER: 20KSPLICE-UPTRACK-MIB::kspliceavailIndex.21 = INTEGER: 21KSPLICE-UPTRACK-MIB::kspliceavailIndex.22 = INTEGER: 22KSPLICE-UPTRACK-MIB::kspliceavailIndex.23 = INTEGER: 23KSPLICE-UPTRACK-MIB::kspliceavailIndex.24 = INTEGER: 24KSPLICE-UPTRACK-MIB::kspliceavailIndex.25 = INTEGER: 25KSPLICE-UPTRACK-MIB::kspliceavailName.0 = STRING: [urvt04qt]KSPLICE-UPTRACK-MIB::kspliceavailName.1 = STRING: [7jb2jb4r]KSPLICE-UPTRACK-MIB::kspliceavailName.2 = STRING: [ot8lfoya]KSPLICE-UPTRACK-MIB::kspliceavailName.3 = STRING: [f7pwmkto]KSPLICE-UPTRACK-MIB::kspliceavailName.4 = STRING: [nxs9cwnt]KSPLICE-UPTRACK-MIB::kspliceavailName.5 = STRING: [i8j4bdkr]KSPLICE-UPTRACK-MIB::kspliceavailName.6 = STRING: [5jr9aom4]KSPLICE-UPTRACK-MIB::kspliceavailName.7 = STRING: [iifdtqom]KSPLICE-UPTRACK-MIB::kspliceavailName.8 = STRING: [6yagfyh1]KSPLICE-UPTRACK-MIB::kspliceavailName.9 = STRING: [bqc6pn0b]KSPLICE-UPTRACK-MIB::kspliceavailName.10 = STRING: [sy14t1rw]KSPLICE-UPTRACK-MIB::kspliceavailName.11 = STRING: [ayo20d8s]KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd]KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k]KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]KSPLICE-UPTRACK-MIB::kspliceavailName.15 = STRING: [qsajn0ce]KSPLICE-UPTRACK-MIB::kspliceavailName.16 = STRING: [5tx9tboo]KSPLICE-UPTRACK-MIB::kspliceavailName.17 = STRING: [2nve5xek]KSPLICE-UPTRACK-MIB::kspliceavailName.18 = STRING: [w7ik1ka8]KSPLICE-UPTRACK-MIB::kspliceavailName.19 = STRING: [9ky2kan5]KSPLICE-UPTRACK-MIB::kspliceavailName.20 = STRING: [zjr4ahvv]KSPLICE-UPTRACK-MIB::kspliceavailName.21 = STRING: [j0mkxnwg]KSPLICE-UPTRACK-MIB::kspliceavailName.22 = STRING: [mvu2clnk]KSPLICE-UPTRACK-MIB::kspliceavailName.23 = STRING: [rc8yh417]KSPLICE-UPTRACK-MIB::kspliceavailName.24 = STRING: [0zfhziax]KSPLICE-UPTRACK-MIB::kspliceavailName.25 = STRING: [ns82h58y]KSPLICE-UPTRACK-MIB::kspliceavailDesc.0 = STRING: Clear garbage data on the kernel stack when handling signals.KSPLICE-UPTRACK-MIB::kspliceavailDesc.1 = STRING: CVE-2011-1160: Information leak in tpm driver.KSPLICE-UPTRACK-MIB::kspliceavailDesc.2 = STRING: CVE-2011-1585: Authentication bypass in CIFS.KSPLICE-UPTRACK-MIB::kspliceavailDesc.3 = STRING: CVE-2011-2484: Denial of service in taskstats subsystem.KSPLICE-UPTRACK-MIB::kspliceavailDesc.4 = STRING: CVE-2011-2496: Local denial of service in mremap().KSPLICE-UPTRACK-MIB::kspliceavailDesc.5 = STRING: CVE-2009-4067: Buffer overflow in Auerswald usb driver.KSPLICE-UPTRACK-MIB::kspliceavailDesc.6 = STRING: CVE-2011-2695: Off-by-one errors in the ext4 filesystem.KSPLICE-UPTRACK-MIB::kspliceavailDesc.7 = STRING: CVE-2011-2699: Predictable IPv6 fragment identification numbers.KSPLICE-UPTRACK-MIB::kspliceavailDesc.8 = STRING: CVE-2011-2723: Remote denial of service vulnerability in gro.KSPLICE-UPTRACK-MIB::kspliceavailDesc.9 = STRING: CVE-2011-2942: Regression in bridged ethernet devices.KSPLICE-UPTRACK-MIB::kspliceavailDesc.10 = STRING: CVE-2011-1833: Information disclosure in eCryptfs.KSPLICE-UPTRACK-MIB::kspliceavailDesc.11 = STRING: CVE-2011-3191: Memory corruption in CIFSFindNext.KSPLICE-UPTRACK-MIB::kspliceavailDesc.12 = STRING: CVE-2011-3209: Denial of Service in clock implementation.KSPLICE-UPTRACK-MIB::kspliceavailDesc.13 = STRING: CVE-2011-3188: Weak TCP sequence number generation.KSPLICE-UPTRACK-MIB::kspliceavailDesc.14 = STRING: CVE-2011-3363: Remote denial of service in cifs_mount.KSPLICE-UPTRACK-MIB::kspliceavailDesc.15 = STRING: CVE-2011-4110: Null pointer dereference in key subsystem.KSPLICE-UPTRACK-MIB::kspliceavailDesc.16 = STRING: CVE-2011-1162: Information leak in TPM driver.KSPLICE-UPTRACK-MIB::kspliceavailDesc.17 = STRING: CVE-2011-2494: Information leak in task/process statistics.KSPLICE-UPTRACK-MIB::kspliceavailDesc.18 = STRING: CVE-2011-2203: Null pointer dereference mounting HFS filesystems.KSPLICE-UPTRACK-MIB::kspliceavailDesc.19 = STRING: CVE-2011-4077: Buffer overflow in xfs_readlink.KSPLICE-UPTRACK-MIB::kspliceavailDesc.20 = STRING: CVE-2011-4132: Denial of service in Journaling Block Device layer.KSPLICE-UPTRACK-MIB::kspliceavailDesc.21 = STRING: CVE-2011-4330: Buffer overflow in HFS file name translation logic.KSPLICE-UPTRACK-MIB::kspliceavailDesc.22 = STRING: CVE-2011-4324: Denial of service vulnerability in NFSv4.KSPLICE-UPTRACK-MIB::kspliceavailDesc.23 = STRING: CVE-2011-4325: Denial of service in NFS direct-io.KSPLICE-UPTRACK-MIB::kspliceavailDesc.24 = STRING: CVE-2011-4348: Socking locking race in SCTP.KSPLICE-UPTRACK-MIB::kspliceavailDesc.25 = STRING: CVE-2011-1020, CVE-2011-3637: Information leak, DoS in /proc.

And here’s what happens after you run uptrack-upgrade -y, using Ksplice to fully upgrade your kernel:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatusKSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: uptodate$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTableKSPLICE-UPTRACK-MIB::ksplicepatchIndex.0 = INTEGER: 0KSPLICE-UPTRACK-MIB::ksplicepatchIndex.1 = INTEGER: 1KSPLICE-UPTRACK-MIB::ksplicepatchIndex.2 = INTEGER: 2KSPLICE-UPTRACK-MIB::ksplicepatchIndex.3 = INTEGER: 3[ . . . ]

The plugin displays that the kernel is now up-to-date.

SNMP and Enterprise Manager

Once the plugin is up and running, you can monitor your system using Oracle Enterprise Manager. Specifically, you can create an SNMP Adapter to allow Enterprise Manager Management Agents to query the status of Ksplice on each system with the plugin installed. Check out our documentation on SNMP support in Enterprise Manager to get started, including section 22.6, “About Metric Extensions”.

This plugin represents the first step in greater functionality between Ksplice and Enterprise Manager and we’re excited about what is coming up. If you have any questions about the plugin or suggestions for future development, leave a comment below or drop us a line at ksplice-support_ww@oracle.com.

[Update]

Notes about Enterprise Manager 12c

When connecting Enterprise Manager 12c (EM) as an SNMP client to the target Ksplice interface, there is an extra step required to fix a known issue where EM is not able to query an SNMP property, e.g:

If your target is “ksplice.example.com”, run the following command on your OMS:

 $ emcli modify_target -name=ksplice.example.com -type=”host” -properties=”SNMPHostname:ksplice.example.com” -on_agent 

That command will populate the missing value of the SNMPHostname property. Then stop and re-start your OMS to make sure the changes take effect.

Ksplice SNMP Plugin

The Ksplice team is happy to announce the release of an SNMP plugin for Ksplice, available today on the Unbreakable Linux Network. The plugin will let you use Oracle Enterprise Manager to monitor the status of Ksplice on all of your systems, but it will also work with any monitoring solution that is SNMP compatible.

Installation

You’ll find the plugin on Ksplice channel for your distribution and architecture. For Oracle Linux 6 on x86_64 that’s ol6_x86_64_ksplice. Install the plugin by running (as root):

yum install ksplice-snmp-plugin

Configuration

If you haven’t set up SNMP before, you’ll need to do a little bit of configuration. Included below is a sample /etc/snmp/snmpd.conf file to try out the plugin:


# Setting up permissions
# ======================
com2sec local localhost public
com2sec mynet 10.10.10.0/24 public

group local v1 local
group local v2c local
group local usm local
group mynet v1  mynet
group mynet v2c mynet
group mynet usm mynet

view all included .1 80

access mynet "" any noauth exact all none none
access local "" any noauth exact all all none

syslocation Oracle Linux 6
syscontact sysadmin <root@localhost>

# Load the plugin
# ===============
dlmod kspliceUptrack /usr/lib64/ksplice-snmp/kspliceUptrack.so

 

You’ll want to replace the IP address next to com2sec with the address of your local network, or the address of your SNMP monitoring software. If you are running on a 32 bit architecture x86), replace lib64 in the dlmod path with lib. The above configuration is an example, intended only for testing purposes. For more information about configuring SNMP, check out the SNMP documentation, including the man pages for snmpd and snmpd.conf.

Examples

You can test out your configuration by using the snmpwalk command and verifying the responses. Some examples:

Displaying the installed version of Ksplice:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceVersion
KSPLICE-UPTRACK-MIB::kspliceVersion.0 = STRING: 1.2.12

Checking if a kernel has all available updates installed:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate

Displaying and comparing the kernel installed on disk with the Ksplice effective version:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceBaseKernel
KSPLICE-UPTRACK-MIB::kspliceBaseKernel.0 = STRING: 2.6.18-274.3.1.el5

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceEffectiveKernel
KSPLICE-UPTRACK-MIB::kspliceEffectiveKernel.0 = STRING: 2.6.18-274.3.1.el5

Displaying a list of all installed updates:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTable

In this case, there are none. This is why the base kernel version and effective kernel version are the same, and why this kernel is out of date.

Displaying a list of updates that can be installed right now, including their description:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable
KSPLICE-UPTRACK-MIB::kspliceavailIndex.0 = INTEGER: 0
KSPLICE-UPTRACK-MIB::kspliceavailIndex.1 = INTEGER: 1
KSPLICE-UPTRACK-MIB::kspliceavailIndex.2 = INTEGER: 2
KSPLICE-UPTRACK-MIB::kspliceavailIndex.3 = INTEGER: 3
KSPLICE-UPTRACK-MIB::kspliceavailIndex.4 = INTEGER: 4
KSPLICE-UPTRACK-MIB::kspliceavailIndex.5 = INTEGER: 5
KSPLICE-UPTRACK-MIB::kspliceavailIndex.6 = INTEGER: 6
KSPLICE-UPTRACK-MIB::kspliceavailIndex.7 = INTEGER: 7
KSPLICE-UPTRACK-MIB::kspliceavailIndex.8 = INTEGER: 8
KSPLICE-UPTRACK-MIB::kspliceavailIndex.9 = INTEGER: 9
KSPLICE-UPTRACK-MIB::kspliceavailIndex.10 = INTEGER: 10
KSPLICE-UPTRACK-MIB::kspliceavailIndex.11 = INTEGER: 11
KSPLICE-UPTRACK-MIB::kspliceavailIndex.12 = INTEGER: 12
KSPLICE-UPTRACK-MIB::kspliceavailIndex.13 = INTEGER: 13
KSPLICE-UPTRACK-MIB::kspliceavailIndex.14 = INTEGER: 14
KSPLICE-UPTRACK-MIB::kspliceavailIndex.15 = INTEGER: 15
KSPLICE-UPTRACK-MIB::kspliceavailIndex.16 = INTEGER: 16
KSPLICE-UPTRACK-MIB::kspliceavailIndex.17 = INTEGER: 17
KSPLICE-UPTRACK-MIB::kspliceavailIndex.18 = INTEGER: 18
KSPLICE-UPTRACK-MIB::kspliceavailIndex.19 = INTEGER: 19
KSPLICE-UPTRACK-MIB::kspliceavailIndex.20 = INTEGER: 20
KSPLICE-UPTRACK-MIB::kspliceavailIndex.21 = INTEGER: 21
KSPLICE-UPTRACK-MIB::kspliceavailIndex.22 = INTEGER: 22
KSPLICE-UPTRACK-MIB::kspliceavailIndex.23 = INTEGER: 23
KSPLICE-UPTRACK-MIB::kspliceavailIndex.24 = INTEGER: 24
KSPLICE-UPTRACK-MIB::kspliceavailIndex.25 = INTEGER: 25
KSPLICE-UPTRACK-MIB::kspliceavailName.0 = STRING: [urvt04qt]
KSPLICE-UPTRACK-MIB::kspliceavailName.1 = STRING: [7jb2jb4r]
KSPLICE-UPTRACK-MIB::kspliceavailName.2 = STRING: [ot8lfoya]
KSPLICE-UPTRACK-MIB::kspliceavailName.3 = STRING: [f7pwmkto]
KSPLICE-UPTRACK-MIB::kspliceavailName.4 = STRING: [nxs9cwnt]
KSPLICE-UPTRACK-MIB::kspliceavailName.5 = STRING: [i8j4bdkr]
KSPLICE-UPTRACK-MIB::kspliceavailName.6 = STRING: [5jr9aom4]
KSPLICE-UPTRACK-MIB::kspliceavailName.7 = STRING: [iifdtqom]
KSPLICE-UPTRACK-MIB::kspliceavailName.8 = STRING: [6yagfyh1]
KSPLICE-UPTRACK-MIB::kspliceavailName.9 = STRING: [bqc6pn0b]
KSPLICE-UPTRACK-MIB::kspliceavailName.10 = STRING: [sy14t1rw]
KSPLICE-UPTRACK-MIB::kspliceavailName.11 = STRING: [ayo20d8s]
KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd]
KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k]
KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]
KSPLICE-UPTRACK-MIB::kspliceavailName.15 = STRING: [qsajn0ce]
KSPLICE-UPTRACK-MIB::kspliceavailName.16 = STRING: [5tx9tboo]
KSPLICE-UPTRACK-MIB::kspliceavailName.17 = STRING: [2nve5xek]
KSPLICE-UPTRACK-MIB::kspliceavailName.18 = STRING: [w7ik1ka8]
KSPLICE-UPTRACK-MIB::kspliceavailName.19 = STRING: [9ky2kan5]
KSPLICE-UPTRACK-MIB::kspliceavailName.20 = STRING: [zjr4ahvv]
KSPLICE-UPTRACK-MIB::kspliceavailName.21 = STRING: [j0mkxnwg]
KSPLICE-UPTRACK-MIB::kspliceavailName.22 = STRING: [mvu2clnk]
KSPLICE-UPTRACK-MIB::kspliceavailName.23 = STRING: [rc8yh417]
KSPLICE-UPTRACK-MIB::kspliceavailName.24 = STRING: [0zfhziax]
KSPLICE-UPTRACK-MIB::kspliceavailName.25 = STRING: [ns82h58y]
KSPLICE-UPTRACK-MIB::kspliceavailDesc.0 = STRING: Clear garbage data on the kernel stack when handling signals.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.1 = STRING: CVE-2011-1160: Information leak in tpm driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.2 = STRING: CVE-2011-1585: Authentication bypass in CIFS.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.3 = STRING: CVE-2011-2484: Denial of service in taskstats subsystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.4 = STRING: CVE-2011-2496: Local denial of service in mremap().
KSPLICE-UPTRACK-MIB::kspliceavailDesc.5 = STRING: CVE-2009-4067: Buffer overflow in Auerswald usb driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.6 = STRING: CVE-2011-2695: Off-by-one errors in the ext4 filesystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.7 = STRING: CVE-2011-2699: Predictable IPv6 fragment identification numbers.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.8 = STRING: CVE-2011-2723: Remote denial of service vulnerability in gro.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.9 = STRING: CVE-2011-2942: Regression in bridged ethernet devices.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.10 = STRING: CVE-2011-1833: Information disclosure in eCryptfs.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.11 = STRING: CVE-2011-3191: Memory corruption in CIFSFindNext.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.12 = STRING: CVE-2011-3209: Denial of Service in clock implementation.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.13 = STRING: CVE-2011-3188: Weak TCP sequence number generation.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.14 = STRING: CVE-2011-3363: Remote denial of service in cifs_mount.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.15 = STRING: CVE-2011-4110: Null pointer dereference in key subsystem.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.16 = STRING: CVE-2011-1162: Information leak in TPM driver.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.17 = STRING: CVE-2011-2494: Information leak in task/process statistics.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.18 = STRING: CVE-2011-2203: Null pointer dereference mounting HFS filesystems.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.19 = STRING: CVE-2011-4077: Buffer overflow in xfs_readlink.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.20 = STRING: CVE-2011-4132: Denial of service in Journaling Block Device layer.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.21 = STRING: CVE-2011-4330: Buffer overflow in HFS file name translation logic.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.22 = STRING: CVE-2011-4324: Denial of service vulnerability in NFSv4.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.23 = STRING: CVE-2011-4325: Denial of service in NFS direct-io.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.24 = STRING: CVE-2011-4348: Socking locking race in SCTP.
KSPLICE-UPTRACK-MIB::kspliceavailDesc.25 = STRING: CVE-2011-1020, CVE-2011-3637: Information leak, DoS in /proc.

And here’s what happens after you run uptrack-upgrade -y, using Ksplice to fully upgrade your kernel:

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatus
KSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: uptodate

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable

$ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::ksplicePatchTable
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.0 = INTEGER: 0
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.1 = INTEGER: 1
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.2 = INTEGER: 2
KSPLICE-UPTRACK-MIB::ksplicepatchIndex.3 = INTEGER: 3
[ . . . ]

The plugin displays that the kernel is now up-to-date.

SNMP and Enterprise Manager

Once the plugin is up and running, you can monitor your system using Oracle Enterprise Manager. Specifically, you can create an SNMP Adapter to allow Enterprise Manager Management Agents to query the status of Ksplice on each system with the plugin installed. Check out our documentation on SNMP support in Enterprise Manager to get started, including section 22.6, “About Metric Extensions”.

This plugin represents the first step in greater functionality between Ksplice and Enterprise Manager and we’re excited about what is coming up. If you have any questions about the plugin or suggestions for future development, leave a comment below or drop us a line at ksplice-support_ww@oracle.com.

[Update]

Notes about Enterprise Manager 12c

When connecting Enterprise Manager 12c (EM) as an SNMP client to the target Ksplice interface, there is an extra step required to fix a known issue where EM is not able to query an SNMP property, e.g:

If your target is “ksplice.example.com”, run the following command on your OMS:

 $ emcli modify_target -name=ksplice.example.com -type=”host” -properties=”SNMPHostname:ksplice.example.com” -on_agent 

That command will populate the missing value of the SNMPHostname property. Then stop and re-start your OMS to make sure the changes take effect.

TEL/電話+86 13764045638
Email service@parnassusdata.com
QQ 47079569