A heap based buffer overflow vulnerability present in KeyView PDF filter as used by Domino can lead to remote arbitrary code execution.
While parsing an ID description of a PDF trailer, no checks are being made regarding its length. The ID, which is usually an ASCII representation of hexadecimal value, is converted into it's byte array equivalent without size checks leading to a straight forward heap overflow which can be abused to overwrite sensitive data structures or heap metadata and subvert the process execution flow to arbitrary code execution.
Shortened test case triggering the vulnerability can be summarized as follows:
%PDF-1.3 %âãÏÓ trailer << /Root 4 0 R /ID[<AAAAAAAAAAAAA...(a large number of hex characters)...41414141 >> startxref 4139 %%EOF
When initializing the parser, a large chunk of memory is allocated in
pdfsr.so library (132648 bytes in size to be precise). An offset into this large context memory
chunk is used as a start of the buffer that ends up being overflown. The overflow happens in the
.text:B79DC7DA loc_B79DC7DA: .text:B79DC7DA mov [ebp+var_15], al  .text:B79DC7DD lea eax, [edx+1]  .text:B79DC7E0 mov [ebp+var_F0], eax  .text:B79DC7E6 movzx eax, byte ptr [edx+1] .text:B79DC7EA mov [ebp+var_14], al  .text:B79DC7ED lea eax, [edx+2] .text:B79DC7F0 mov [ebp+var_F0], eax .text:B79DC7F6 mov [ebp+var_13], 0 .text:B79DC7FA mov dword ptr [esp+0Ch], 0 ; group .text:B79DC802 mov dword ptr [esp+8], 10h ; base .text:B79DC80A mov dword ptr [esp+4], 0 ; endptr .text:B79DC812 lea eax, [ebp+var_15] .text:B79DC815 mov [esp], eax ; nptr .text:B79DC818 call ___strtol_internal  .text:B79DC81D mov edx, [ebp+counter] .text:B79DC823 mov [edx+edi+11D8h], al  .text:B79DC82A mov edx, [ebp+var_F0] .text:B79DC830 cmp esi, edx  .text:B79DC832 jbe loc_B79DCB16 .text:B79DC838 add [ebp+counter], 1 .text:B79DC83F movzx eax, byte ptr [edx] .text:B79DC842 cmp al, '>'  .text:B79DC844 jnz short loc_B79
At the beginning of the above code, at ,
al contains the first character of the ID hex string
which is saved in
var_15, at , and  the second character is saved in
var_14 which, along
var_15, makes up a string that is passed to
strtol at . Function
strtol is called
with base set to 0x10, converting from hex string into bytes. At  the returned value is written
to the buffer. This is where the out of bounds write occurs when the counters are past the boundary
of the buffer. At  the current string counter is checked against the file end,
to the end of file. If the check at  passes, another check is made at  which is looking
for a terminating
> character. No buffer length checks are being made.
This buffer overflow can be abused to overwrite other parts of the context structure or even, further up the heap, the heap metadata thus leading to process crash and possible arbitrary code execution.
The supplied test case is crafted in such way that the buffer overflow will overwrite a pointer on the heap which is later dereferenced during heap cleanup. The process crashes while calling free on an invalid pointer. Further control over data overwritten in the context structure can lead to arbitrary code execution.
Detection of PDF files specifically crafted to trigger this vulnerability can be based on the abnormally large length of the ID value in the PDF trailer.
The vulnerability can be triggered with the supplied test case in the
filter standalone KeyView
binary shipped with IBM Domino, or by sending it as an attachment with an email to a Domino
2016-02-09 - Vendor Notification
2016-06-08 – Public Disclosure
Discovered by Aleksandar Nikolic of Cisco Talos.