Comprehensive comparison and evaluation of domestic AI programming assistant and GitHub Copilot integrated with GPT-4! Who is better? The result turned out to be...
Latest update time：2024-01-09
Reads：
"
Who is better, Code Raccoon or GitHub Copilot? Find out through a comprehensive comparison and evaluation!
"
With the popularity of big models, AI programming assistant products for developers are now blooming everywhere.
However, in the eyes of a "layman" like the editor, the differences between these programming assistants are not very obvious, and the code generation effect passing by line by line is not intuitive, unless...
AI, you come directly to the talent show. It’s New Year, let’s perform a show for everyone.
Don’t tell me, there really is such a brave AI——
SenseTime released a video in which a novice developer used their "Raccoon" product to implement a "holiday game" with complete code from scratch.
At present, there are many such code programming assistant products in China, but there are absolutely few that directly announce the complete development and application process like SenseTime Xiaohuixiong. You see, Santa Claus is quite cute when he jumps around.
But don’t be “confused”. The Google Gemini new product video that amazed everyone at the time was also questioned for exaggerating the AI performance through editing. Fortunately, the official public beta link is listed below the little raccoon’s “performance” video:
"Code Raccoon" is now open for public testing. Welcome to visit: https://code.sensetime.com to register and experience it.
Highlights: 30+ mainstream programming languages such as Python, Java, JavaScript, C++, Go, and SQL, as well as mainstream IDEs (integrated development environments) such as VS Code and IntelliJ IDEA, are all supported.
The editor decided to invite a professional developer friend to evaluate the key functions presented in the video.
The strong must challenge the stronger. According to public information, the performance of Code Little Raccoon in the professional list has exceeded GPT-3.5, so this time I chose another highly popular intelligent programming assistant
based on "The Strongest on the Surface" "GPT-4's Github Copilot
is undergoing simultaneous evaluation. It coincides with the update of GitHub Copilot in the past few days. You can take a look at its strength together.
Speaking of Chinese understanding ability, there is indeed a "Party A briefing" link at the beginning of the video. The operation novice allowed Code Raccoon to clearly realize the output process from game idea to complete planning using only natural language interaction. It is Really starts from 0.
OK, the conversation ends here.
Talk is cheap, show me the code.
The following is a summary of the evaluation:
01
Code Raccoon is more "cost-effective"
Code functions are evenly divided. Complex scenes still have to look at "the strongest on the surface"
Let me start with the conclusion: each has its own merits.
The advantage of SenseTime's Little Raccoon Code Assistant is that it is completely free for individual developers
(it's not that Copilot cannot afford it, Code Little Raccoon is more cost-effective)
, does not require redundant network configuration, and is more convenient to use; it provides strong support in the Chinese community , including a dedicated Chinese Q&A group and rapid account approval service, is a
programming assistant
more suitable for Chinese developers
.
In terms of code functions, the code generation performance of Code Little Raccoon is quite close to GPT-4, but there is a gap with GPT-4 in terms of general dialogue. In addition, the deep integration of GitHub Copilot and GitHub enables it to fully optimize the user's development process.
But it requires a fee, which is $10 per month.
But in more complex scenarios, GitHub Copilot performs better and understands complex tasks more thoroughly.
Let’s look at a detailed comparison of code functions:
Code completion: Code Raccoon 4 stars VS GitHub Copilot 4 stars
For this functional test, I came up with a classic question: write a python function for quick sorting and add test cases.
The code completion speed of Little Raccoon is relatively fast, and the great thing is that it can be run directly without debugging, which is quite worry-free.
The implementation effect of Github Copilot is similar to that of Code Raccoon.
Swipe up and down to view the long image
Code Translation: Code Raccoon 4 stars VS GitHub Copilot 4 stars
The question is to translate the code of the previous question into golang language. The two assistants are equally matched in this function. They can both complete it quickly and run through it in one go. The following is the code translated by the two:
Code raccoon:
Github
C
op
i
lo
t
:
Swipe up or down to
view
the long image
Code correction: Code Raccoon 3.5 stars VS GitHub Copilot 4 stars
It’s time for error correction. Let’s be a little naughty and deliberately write two errors to see if they can find bugs. Can everyone spot it?
Announce the answer in advance:
The first bug is in line 8. I replaced the > symbol with a < symbol.
The second bug is line 9. I exchanged the order of quick_sort(right) and middle.
The code raccoon only found the first bug. It seems that I need to remind you that the world still cannot function without us humans. I posted the printed results of the unit test to the little raccoon. This time, I found the second bug. .
This time, he also kindly added comments where the errors were corrected, which is very praiseworthy!
This time Github Copilot performed relatively better and could find all errors after just one modification.
Code refactoring: Code Raccoon 4 stars VS GitHub Copilot 4 stars
The challenge continues. Next, I ask them to refactor this function to save memory overhead as much as possible, because the arr array passed in may be very large, and at the same time, it is required to improve the readability of the code.
AI also knows how to reduce costs and increase efficiency. It uses memory optimization of in-place sorting to reduce resource overhead. At the same time, some code blocks are abstracted into sub-functions, so the code is relatively elegant and easy to understand.
Also let GitHub
C
op
i
lo
t
refactor the following code, the effect is also good, the implementation method is basically similar to the code raccoon version.
Unit testing: Code Raccoon 4 stars VS GitHub Copilot 3.5 stars
Good unit testing is very important for code correctness and quality. Fortunately, Little Raccoon provides the function of automatically generating unit tests.
I tried to add some more test cases to it, which can randomly generate arrays and require relatively large data, such as 10,000 elements, to see the sorting accuracy and speed.
It can be seen that this function is still very convenient, because the test code of Little Raccoon is written very quickly, and the correctness also meets my two requirements, which is to add random array generation and measure the sorting time.
I like this feature very much, give it a thumbs up.
Github
C
op
i
lo
t
also has a similar function, but I personally think it is slightly inferior to Little Raccoon. The code generated for the first time has no printing time, and it can be printed after the second prompt.
Swipe up or down to
view
the long image
In addition to the above code functions, Code Raccoon also provides many small functions that are convenient to improve developer efficiency, such as writing commit messages and automatically generating commit messages based on my code changes this time. This is quite easy to use. I like it very much. My mother no longer has to worry about me not being able to write commit messages.
There is also a function called "favorite", which allows you to directly save your favorite code snippets for subsequent use.
Github
C
op
i
lo
t
also has another considerate feature. Every time the code is generated, there will be recommendations for related issues, such as some new performance optimization suggestions for the currently generated code.
Generally speaking, although each has its own merits, SenseTime Code Raccoon is more balanced in terms of functional capabilities, especially showing its advantages in adapting to Chinese developers.
Finally, let’s look at the performance of the two in complex scenarios, taking the understanding and implementation of complex business logic as an example. Because we often need to implement complex business logic, the prerequisite for writing correct code is to understand the business logic itself, so let’s look at Look at the code assistant's ability to understand complex business logic.
Taking financial business as an example, the topic is as follows:
There are some problems with the code logic in Little Raccoon's reply, such as the required function signature being incorrect.
Swipe up or down to view the long image
In comparison, Github Copilot's response is clearer. Of course, readers are welcome to personally review whether Copilot's effect is correct.
02
Real business development needs are complex and diverse
Code Raccoon has achieved comprehensive and effective assistance
Okay, this review has come to an end for now. Which ones are strong and which are weak, and the user experience will vary depending on individual differences, but one thing is certain: the current code assistants do play a significant role in simplifying programming tasks, although they also face common challenges.
The reason why Github Copilot is better than most domestic AI programming assistants in dealing with complex scene evaluations is that, in addition to using GPT-4, the "strongest" large model on the earth's surface, as the bottom layer, it is also largely attributed to Github Copilot's performance in these scenarios. Specially optimized.
To practice in real business scenarios with "real swords and guns"
may be the next methodology for all this type of products.
Taking the product under this review as an example, Code Raccoon is currently in open beta and is open to developers from all walks of life and with different programming abilities for trial use.
Various problems will be encountered during the programming process, and business needs are also different:
Relatively
elementary
programming syntax
issues, using a certain language to implement specific functions. For example, for UI/UX designers who need to implement their designs into actual front-end code, they use code raccoons to understand and implement complex CSS and JavaScript logic to ensure their designs render accurately on the web page.
Or newbies who are more entry-level will also be
trapped by
the concept explanation type
questions when
learning how to use Python to perform file operations, such as reading and writing files. They start writing code but hit a wall. By interacting with Code Raccoon, they were provided with code samples and explanations to help them understand basic concepts of file operations.
For those developers with rich work experience, they need code assistants to provide some practical solutions. An architect shared that they use Code Raccoon to explore and compare the advantages and disadvantages of different programming languages and frameworks, so as to choose the best solution for the project. Suitable technical solutions.
Of course, there are more scenarios in actual business that are far more complex than these cases, but Code Raccoon can now play an effective assisting role for developers in different fields, different development scenarios, and with different experiences. .
Our development evaluation this time tried to be more comprehensive, but the limitations of energy and resources are far from enough for actual development needs, and these products will also need to be continuously upgraded in the future.
It is understood that in the future, with the continuous enrichment of SenseTime's "Daily New SenseNova" large model system, Raccoon will continue to evolve and gradually introduce code warehouse-level memory, AI Agents and other capabilities to further improve programming efficiency and development. experience.
The best alchemy stone is the actual use of developers. If you have any questions, you are welcome to click
to read the original text
and try it~
//
Recent popular articles
京公网安备 11010802033920号