Type Casting

Type Casting

There was an interesting question in the Vulnerability Research lecture regarding the type casting example. More specific the question was about the casting and the output after the casting from foo_t to bar_t (see code below). We’d like to shed some light on this.

snippet of the relevant code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
        foo_t f;
        f.bank_account = 1337;
        f.balance = 1.234567;

        printf("f.bank_account: %d\n", f.bank_account);
        printf("f.balance: %f\n", f.balance);

        // cast foo_t to bar_t
        bar_t *g = (bar_t *)(&f);

        printf("g.bank_account: %d\n", g->bank_account);
        printf("g.balance: %f\n", g->balance);

output

1
2
3
4
5
6
7
8
9
hello!!!
say_hello_ptr: 0x401136
36 11 40 00 
00401136
6@
f.bank_account: 1337
f.balance: 1.234567
g.bank_account: 4202574
g.balance: 0.000000

https://godbolt.org/z/aa1c7YMd8

The output might change with different compiler flags such as (-O0, -O1, -O2 and others).

analysis

The following code is from Ghidra. We added some comments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
      ////
      // THIS IS THE VALID PART
      ////
      
      // store 1337 on the stack (inside the struct)
      .text:0040123c c7 45 d0 39 05 00 00    MOV        dword ptr [RBP + local_38],0x539
      
      // store 1.234567 on the stack (inside the struct; indirection via XMM0 floating point register)
      .text:00401243 f3 0f 10 05 dd 0f       MOVSS      XMM0,dword ptr [DAT_00402228]    = 3F9E064Bh
                     00 00
      .text:0040124b f3 0f 11 45 d4          MOVSS      dword ptr [RBP + local_34],XMM0
      
      // set integer value for %d in printf (ESI)
      .text:00401250 8b 45 d0                MOV        EAX,dword ptr [RBP + local_38]
      .text:00401253 89 c6                   MOV        ESI,EAX
      
      // format string address goes to EDI
      .text:00401255 bf e1 21 40 00          MOV        EDI,s_f.bank_account:_%d_004021e1    = "f.bank_account: %
      
      // EAX is set to 0: zero floating point registers used in printf's arguments
      .text:0040125a b8 00 00 00 00          MOV        EAX,0x0
      .text:0040125f e8 dc fd ff ff          CALL       <EXTERNAL>::printf    int printf(char * __
      
      // set float value for %f
            
      // move float value from stack to XMM0
      .text:00401264 f3 0f 10 45 d4          MOVSS      XMM0,dword ptr [RBP + local_34]
      
      // set XMM1 to 0
      .text:00401269 66 0f ef c9             PXOR       XMM1,XMM1
      
      // convert singe precision float to double precision float (as printf expects double precision)
      .text:0040126d f3 0f 5a c8             CVTSS2SD   XMM1,XMM0
      
      // move float through registers to finally be in XMM0 (if there are more float args they would go to XMM1, ...)
      .text:00401271 66 48 0f 7e c8          MOVQ       RAX,XMM1
      .text:00401276 66 48 0f 6e c0          MOVQ       XMM0,RAX
      
      // format string address goes to EDI
      .text:0040127b bf f5 21 40 00          MOV        EDI,s_f.balance:_%f_004021f5    = "f.balance: %f\n"
      
      // now EAX is set to 1 because we have one floating point argument passed to printf("...format string...", arg_1, arg_2, ...)
      .text:00401280 b8 01 00 00 00          MOV        EAX,0x1
      // do the call to printf
      .text:00401285 e8 b6 fd ff ff          CALL       <EXTERNAL>::printf    int printf(char * __
      
      ////
      // THIS IS THE INVALID PART
      ////
      
      // NOW the casting happens in the code!
      
      // get local_38 which is our integer! and move it to XMM0; at local_28 will be a pointer to local_38
      // here it gets interesting, we prepare the float argument although we wanted to use it with %d
      // mostly the same instructions as above (XMM2 instead of XMM1)
      
      .text:0040128a 48 8d 45 d0             LEA        RAX=>local_38,[RBP + -0x30]
      .text:0040128e 48 89 45 e0             MOV        qword ptr [RBP + local_28],RAX
      .text:00401292 48 8b 45 e0             MOV        RAX,qword ptr [RBP + local_28]
      .text:00401296 f3 0f 10 00             MOVSS      XMM0,dword ptr [RAX]=>local_38
      .text:0040129a 66 0f ef d2             PXOR       XMM2,XMM2
      .text:0040129e f3 0f 5a d0             CVTSS2SD   XMM2,XMM0
      .text:004012a2 66 48 0f 7e d0          MOVQ       RAX,XMM2
      
      // integer will land in XMM0 finally but as a float representation
      .text:004012a7 66 48 0f 6e c0          MOVQ       XMM0,RAX
      
      // format string address goes to EDI
      .text:004012ac bf 04 22 40 00          MOV        EDI,s_g.bank_account:_%d_00402204    = "g.bank_account: %
      
      // now EAX is set to 1 because we have one floating point argument passed to printf("...format string...", arg_1, arg_2, ...)
      .text:004012b1 b8 01 00 00 00          MOV        EAX,0x1
      
      // to the call to printf BUT
      // although we prepared a floating point register (XMM0) printf sees that there is a %d and looks for it where an integer argument should be: in RSI/ESI (calling convention RDI, RSI, RDX, RCX, R8, R9; return value in RAX)
      // SO: RSI is still the value that has been set above, more concrete, the value 0x4052a0 or 4215456 which is the address of a format string -> so completely unrelated to the data, we wanted to print
      .text:004012b6 e8 85 fd ff ff          CALL       <EXTERNAL>::printf    int printf(char * __
      
      // here we will print an integer argument... but we used %f for floats
      // now it is the other way arround: we copy the float value from the stack via EAX to ESI (normally for the first %d)
      // BUT since we us %f in the format string printf will look for this value in XMM0 which in our case is set to 0 from previous operations (again unrelated to what we wanted to actually print)
      .text:004012bb 48 8b 45 e0             MOV        RAX,qword ptr [RBP + local_28]
      .text:004012bf 8b 40 04                MOV        EAX,dword ptr [RAX + local_34]
      .text:004012c2 89 c6                   MOV        ESI,EAX
      .text:004012c4 bf 18 22 40 00          MOV        EDI,s_g.balance:_%f_00402218    = "g.balance: %f\n"
      .text:004012c9 b8 00 00 00 00          MOV        EAX,0x0
      .text:004012ce e8 6d fd ff ff          CALL       <EXTERNAL>::printf    int printf(char * __

summary

Probably not the best example for type casting but you can see here that things happen that are not expected by the programmer. It is just how printf works internally and which registers are expected for floats (XMM0, XMM1, ...) or for, e. g., ints (RDI, RDX, RCX, ...).

material