AI based code review – Using ChatGPT to capture a flag | Daily News Byte

[ad_1]

In this blog, we will share our experience using ChatGPT in the Aptos CTF MOVEment 2022 competition and discuss the benefits of using this tool in CTF competitions.

Recently, CertiK participated in the Aptos CTF MOVEment 2022 competition. As with all Capture the Flag (CTF) competitions, we were challenged to quickly understand and summarize the semantics of complex code snippets. To tackle this challenge, we turned to ChatGPT, a large language model trained by OpenAI. ChatGPT proved to be a valuable tool in this regard, as it was able to understand and explain code written in different programming languages.

First impressions

By interacting with ChatGPT through natural language, we were able to ask it questions about the code and get clear and concise answers. This allowed us to quickly understand the functionality of the code, saving us valuable time during the CTF competition. For example, when presented with a code snippet written in Move, ChatGPT can provide a summary of the code’s semantics and explain what the code does in plain, natural language. This can save us a lot of time and effort when trying to quickly understand the code during a CTF competition.

Example 1: Summarizing code functionality

Here is an example of how ChatGPT can summarize the functionality of this code snippet provided in the CTF challenge:

public fun pow(g: u128, x: u128, p: u128): u128 {
let ans = 1;
g = g % p;
while (x !=0) {
if ((x & 1) == 1) {
ans = ((ans % p) * (g % p)) % p;
};
x = x >> 1;
g = (g * g) % p;
};
ans
}

ChatGPT summary: The code defines a public function called pow which requires three parameters: g, xand p. The function calculates and returns the result of the lift g to the kth power, modulo p. It does this by using a loop for multiple times in the square g and multiply by ans if it is the least significant part x is 1. The result is then returned. Overall, the code provides a way to calculate modular exponentiation g to the kth power, modulo p.

We were very impressed with the results we got using ChatGPT. Despite the fact that the code was not particularly long, it was much faster to understand and summarize with the help of ChatGPT than by reading the code directly. This illustrates the potential value that ChatGPT can provide for inexperienced developers. Using ChatGPT, they can quickly get an overview of the code and understand its purpose and structure, which can save them a lot of time and effort.

Digging deeper

Although MOVE is a relatively new language, ChatGPT was able to provide us with useful information. We decided to further test its capabilities by trying a slightly more advanced program. Here’s another example where ChatGPT can help explain corner cases that lead to runtime failures and help us debug during code testing.

Example 2: ChatGPT can help locate a potential runtime error

Q: In what cases would the following code not execute?

fun encrypt_string(plaintext : vector<u8>) : vector<u8> {
let key : vector<u8> = transaction_context::get_script_hash();
let key_len : u64 = vector::length(&key);let ciphertext : vector<u8> = vector::empty<u8>();
let i = 0;
while ( i < vector::length(&plaintext) ) {
vector::push_back(&mut ciphertext, *vector::borrow(&plaintext, i) ^ *vector::borrow(&key, (i % key_len)));
i = i + 1;
};
ciphertext
}

ChatGPT’s response: The code in question may fail to execute if transaction_context::get_script_hash() returns an empty vector or if vector::length() coming back 0 when referring to a plaintext or key. In either of these cases, the while loop in the code would not execute, and the function would simply return an empty vector.

While ChatGPT may not be able to identify the exact location of the exception, it can still provide useful insights and descriptions of angles and potential problems. In this case, ChatGPT may be able to suggest that transaction_context::get_script_hash() function can return an empty vector, which can cause an error when used in an operation such as i % key_len. This information can be useful for someone trying to debug code and understand potential sources of errors.

Handling Move Bitecode

Apart from summarizing the semantics of the code, we were also interested in whether ChatGPT could understand scroll bytecode when represented in different formats. We found that ChatGPT can understand Move bytecode when written in text, but is unable to interpret raw bytecode instructions and provide information about their meaning or behavior.

Q: Can you explain the following bytecode?

FunctionDefinition { function: FunctionHandleIndex(1), visibility: Private, is_entry: false, acquires_global_resources: [], code: Some(CodeUnit { locals: SignatureIndex(9), code: [Call(11), StLoc(3), ImmBorrowLoc(3), CallGeneric(0), ImmBorrowLoc(3), VecLen(11), StLoc(4), VecPack(11, 0), StLoc(1), LdU64(0), StLoc(2), CopyLoc(2), ImmBorrowLoc(0), VecLen(11), Lt, BrFalse(35), Branch(17), MutBorrowLoc(1), ImmBorrowLoc(0), CopyLoc(2), VecImmBorrow(11), ReadRef, ImmBorrowLoc(3), CopyLoc(2), CopyLoc(4), Mod, VecImmBorrow(11), ReadRef, Xor, VecPushBack(11), MoveLoc(2), LdU64(1), Add, StLoc(2), Branch(11), MoveLoc(1), Ret] }) },

Summary of ChatGPT: The code you provided is a function definition written in Move bitcode. A function is defined by a FunctionDefinition block, which specifies functions function handle, its visibility (in this case, Private), is it the entry point (false in this case), the global resources it acquires (none in this case) and the function code.

The function code itself is a sequence of bytecode instructions, each of which performs a specific operation. For example, the Call the instruction calls the function, i.e StLoc instruction stores the value in a local variable, the ImmBorrowLoc an instruction invariably borrows a local variable, and so on.

It is difficult to give a detailed explanation of the functionality of the code without knowing more about the context in which it is used and the meanings of the various instructions and values. However, in general, the code appears to define a function that calls other functions, manipulates local variables, and performs some arithmetic and logical operations. It also uses vector operations, such as VecLen and VecPushBack, and borrows the local variables immutably and mutable. Overall, the code seems complex and would need more information to understand its exact functionality.

This outcome is somewhat expected. ChatGPT fails to provide an insightful understanding of the code. This is understandable because ChatGPT is a language model and is not specialized enough to understand Move bitcode (yet). However, it still generates a meaningful description for a given program.

Restrictions

However, it is important to note that ChatGPT has its limitations. It can understand the basics of Move source code and provide a general explanation of its semantics, but it is unable to perform detailed logic simulations or provide reliable information about code behavior. In other words, ChatGPT can be a useful tool for quickly understanding code written in Move, but should not be relied upon for more advanced tasks or for detailed analysis of code behavior.

Example 4: Failed to parse code for target branch

Q: Given Initialize_balance is set to 10 and the initial value of res.balance is set to 10. How can we call a function to allow res.k3 to be true? (Assume we can call the function multiple times)

public entry fun add(account: &signer, choice: u8, number: u8) acquires Challenge {
let res = borrow_global_mut<Challenge>(signer::address_of(account));
assert!(number <= 5, 0);
if (choice == 1) {
res.balance = res.balance + number;
} else if (choice == 2) {
res.balance = res.balance * number;
} else if (choice == 3) {
res.balance = res.balance << number;
};if (!res.q3 && res.balance < Initialize_balance) {
res.q3 = true;
}
}

ChatGPT’s response: To call add function and set res.q3 to the trueyou can call the function with choice set to 3 and number set to a value that causes res.balance to be less than Initialize_balance after res.balance is shifted to the left by significant numbers.

For example, if the initial value is from res.balance is 10you can call the function with the selection set to 3 and the number set to 3which will move res.balance left by 3 bits, resulting in a value of 80. Since this is greater than Initialize_balancewhich is set to 10the if condition in the addition function will be false and res.q3 it will remain false.

However, if you call the function with choice set to 3 and number set to 2the res.balance will be shifted left by 2 bits, resulting in a value of 40. Since this is less than Initialize_balancethe if condition in the addition function will be true and res.q3 will be set to true.

Therefore, to call the function and set res.q3 on true, you can call the add function with choice set to 3 and number set to a value that causes res.balance to be less than Initialize_balance after res.balance is moved to the left by a number.

In this case, ChatGPT fails to give the correct answer, trying to claim that 40 is less than 10.

Conclusion

In summary, our experience using ChatGPT to resolve CTFs has been positive. ChatGPT’s ability to understand and summarize code written in Move is particularly helpful. Although it has its limitations, we believe that ChatGPT can be a useful tool for developers working with this language.

By the way, in addition to the answers provided by ChatGPT, part of this story is written with text generated by ChatGPT. If we include the responses generated by ChatGPT, close 90% of the content of this blog is generated by ChatGPT. Although still not perfect, ChatGPT is an amazing app.

This article is a press release.

What is Certik:

Certik is a blockchain security firm that helps projects identify and eliminate security vulnerabilities in blockchains, smart contracts, and Web3 applications using its cybersecurity services, products, and techniques.