Implicit int and Pointers
So here is an interesting problem related to C that may seem completely innocuous but can blow up in your face. This is something I see a lot with students who are used to Java but not C. For this example, I am going to focus on a x86-64 Linux system. Let's say you have a function that returns a pointer and is defined in one file and used in another. Specifically, you'd have the following three files:
my_struct.h:
#ifndef _MY_STRUCT_H_ #define _MY_STRUCT_H_ struct my_struct { int foo; int bar; }; #endif // _MY_STRUCT_H_
main.c:
#include <stdio.h> #include "my_struct.h" int main(void) { struct my_struct *instance = alloc_my_struct(5, 10); printf("instance in %s = %p\n", __func__, (void *)instance); // rest of code ... instance->foo = 10; return 0; }
my_struct.c:
#include <stdio.h> #include <stdlib.h> #include "my_struct.h" struct my_struct* alloc_my_struct(int foo, int bar) { struct my_struct *instance = malloc(sizeof(*instance)); if(instance != NULL) { instance->foo = foo; instance->bar = bar; } printf("instance in %s = %p\n", __func__, (void *)instance); return instance; }
I should preface by saying that I generally perfer a pattern more like:
int init_my_struct(struct my_sturct *instance, int foo, int bar)
with the return value indicating success or failure, if appropriate. This way, the memory for the struct can be allocated statically, dynamically, or
automatically but let's continue with the previous example.
So, when you compile
main.c, you get a warning from gcc (version 4.9.2):
main.c:7:32: warning: initialization makes pointer from integer without a cast struct my_struct *instance = alloc_my_struct(5, 10);
but when you run the executable, the output shows the same value for the pointer so everything must be good (of course, it isn't, otherwise this would be boring).
instance in alloc_my_struct = 0x23a5010 instance in main = 0x23a5010
Now, let's say you modify
struct my_struct which results in its size being increased:
my_struct.h:
#ifndef _MY_STRUCT_H_ #define _MY_STRUCT_H_ struct my_struct { int foo; int bar; char large_buffer[0x1000000]; }; #endif // _MY_STRUCT_H_
Now, because the size of
struct my_struct has increased, when you run the simple test program, you get something like:
instance in alloc_my_struct = 0x7fdff615f010 instance in main = 0xfffffffff615f010 Segmentation fault (core dumped)
or something like:
instance in alloc_my_struct = 0x7f8d35e55010 instance in main = 0x35e55010 Segmentation fault (core dumped)
What went wrongWhat went wrong
I'm changing the struct size so the address returned by
malloc is higher, which is the key thing that has changed. So the question is, what happened here? Well it probably had to do with the fact that initialization makes pointer from integer without a cast warning and that warning happened because we did not provide a function prototype. Specifically, you can get one more warning if you compile with
-Wall.
main.c:7:10: warning: implicit declaration of function ‘alloc_my_struct’ [-Wimplicit-function-declaration] struct my_struct *instance = alloc_my_struct(5, 10); main.c:7:32: warning: initialization makes pointer from integer without a cast struct my_struct *instance = alloc_my_struct(5, 10);
So let's add a function prototype to
my_struct.h:
#ifndef _MY_STRUCT_H_ #define _MY_STRUCT_H_ struct my_struct { int foo; int bar; }; struct my_struct *alloc_my_struct(int foo, int bar); #endif // _MY_STRUCT_H_
Now, you don't get any compiler warnings and the program output is something like the following:
instance in alloc_my_struct = 0x7f83c47a3010 instance in main = 0x7f83c47a3010
Looking at some assemblyLooking at some assembly
So, we're all good again. But what exactly is happened when we included the function declaration? Well, as the warning states, there is a conversion from an integer to a pointer. But where is the integer? As the name of this post implies, the integer is implicit. When gcc encounters a function invocation that it has not seen the declaration for, it assumes that the function has a variable number of arguments, and that the return type is an
int. So let's look at the relevant snippet of assembly to see the exact difference for when the function prototype is included or not. To do this, we will use the
-S flag of gcc that outputs the assembly.
With the function prototype:
subq $16, %rsp movl $10, %esi movl $5, %edi call alloc_my_struct movq %rax, -8(%rbp) movq -8(%rbp), %rax movq %rax, %rdx
Without the function prototype:
subq $16, %rsp movl $10, %esi movl $5, %edi movl $0, %eax call alloc_my_struct cltq movq %rax, -8(%rbp) movq -8(%rbp), %rax movq %rax, %rdx
So, the difference is that when the function prototype is not included, you get a
movl $0, %eax before the
call instruction and you get a
cltq after. The
movl $0 %eax isn't relavent to this discussion, so you can find the reason behind it at the end of this post. The interesting part is the
cltq instruction.
ctlq (which is the AT&T name for the
cdqe instruction) sign extends the dword in
%eax to a qword, storing the result in
%rax. Since gcc is assuming the return value is a 32 bit integer, it is sign extending it to fit within the 64 bit pointer variable. This is in fact the problem.
If you look at the case where the pointer did not change, the value returned by
malloc has bits 31 to 63 set to 0. Therefore, the sign extension didn't change
%rax. In the two cases where the pointer changed, bits 32 to 63 are set to all 0's or all 1's, depending on the value of bit 31.
Setting %eax to 0 before the function callSetting %eax to 0 before the function call
The
movl $0, %eax before the
call happens because gcc assumed that
alloc_my_struct takes a variable length number of arguments. This is, again, because the prototype has not been specified. The AMD64 calling convention specifies that the number of vector arguments in the variable length argument list is passed via register
%rax. Vector arguments are arguments that use the SEE or AVX (Advanced Vector Extensions) registers. In this case, we have just two integers passed via the general purpose registers so the value is 0.
Final ThoughtsFinal Thoughts
So what have we learned? Well, the obvious one is don't ignore warnings, but also that something as simple as a C function return value can work sometimes and fail other times when you ignore them. The code for this post can be found here.
This post was originally posted by the author on his blog, Ring Zero and Lower. This version has been edited for clarity and may appear different from the original post.