What happens when a company introduces GitHub Copilot into their daily workflow?
Quick Overview of Key Insights
- Our active experimentation with GitHub Copilot aims to understand the practical benefits and potential enhancements in productivity this tool might bring to active missions.
- This real-world application testing will yield more comprehensive and exciting results for companies considering its implementation in their daily workflow.
- Copilot’s non-intrusive nature is a pivotal advantage as it supplements the workflow without causing major disruptions.
- Despite expectations that it would be disruptive or of limited use, Copilot seems promising that it can enhance productivity by automating mundane and labor-intensive tasks, thereby allowing engineers to devote more time to complex and creative work
The Potential of AI Augmentation in Software Engineering
Too often, the use cases of new AI tools are done in a bit of a vacuum, isolated spaces that do not reflect everyday workflows. As Nicholas Brochu, Head of Data Engineering at Mission puts it, “Who cares if it can do a little magic trick in a blank file, right?”
Here at Mission, we were curious how effective one such tool, GitHub Copilot, could be if it were used on active missions. We searched online for articles documenting this sort of thing and came up with too few. So, to supply our software engineering community with the kind of content we had trouble finding ourselves, we’ve decided to keep a log of sorts as we work with Copilot, sharing our opinions and any useful data we gather along the way.
Will Copilot enhance the productivity of our developers or become a distraction? Will its suggestions be useful or lead to frustration over their uselessness?
At Mission, we aim to level up engineers and push software engineering to new heights. So when a tool as powerful as AI augmentation comes along, we can’t help but engage with it with a sincere enthusiasm to further our cause. For us, it’s not about hopping on the bandwagon just to avoid FOMO. We’re constantly experimenting to discover the ideal software engineering environment to yield the best possible results. This experiment is a natural expression of this core ambition of ours.
In the first installment of what will be an ongoing series, with updates as we have them for you, we’ve decided to outline some of the basics of our expectations, some very early opinions, and speculations as to how AI might augment the work we do.
Unpacking the Use Application of Copilot
Copilot has the potential to act as a diligent partner to software engineers, offering insights and suggestions throughout the coding process which might be accepted with a simple tab. Leveraging machine learning algorithms, it scrutinizes the code being crafted in real-time, identifying patterns, and providing intelligent recommendations for improvements or completions. These propositions materialize as unobtrusive gray text within the code editor, empowering the engineer to either integrate or disregard the suggestion.
By shouldering the burden of repetitive and meticulous tasks, Copilot potentially paves the way for engineers to allocate their mental resources towards more intricate and creative pursuits. This ingenious tool promises a paradigm shift in productivity, transforming the traditional landscape of software engineering.
Real-world Experiments & Anticipated Benefits
A recent survey of the effects of AI adoption on company productivity had companies reporting “that they can work more than 2.5x faster (a 155% increase in their software development lifecycle speed)” while “some engineers have even reported 11x (1000%) gains.”
In another earlier study (September 2022) conducted by GitHub themselves, one which collected over 2,000 responses and focused more on engineer satisfaction and overall experience, 88% of respondents indicated that they had faster completion times, while 96% reported that they’d become faster with repetitive tasks, with similarly high percentages (75%+) for being “more in the flow” with “less time searching” and “less mental effort on repetitive tasks. Overall, the majority of the respondents felt less frustrated when coding, more fulfilled with their job, and felt able to focus on “more satisfying work”.
However, the details of the specific contexts in which this information was gathered varies widely, and in some cases is not available in greater detail. Our exploration into the potentialities of Copilot thus diverges from conventional AI writeups.
Instead of confining the technology to isolated environments, we are deliberately deploying Copilot within our ongoing missions, which we will be able to report in much more specific detail. The ultimate goal is to validate its potential in practical, real-world applications and to draw a much more comprehensive understanding of its capacity to boost productivity.
A salient advantage of Copilot lies in its non-intrusive nature. It seamlessly merges with the engineer’s workflow, offering assistance without necessitating any disruptive changes. Furthermore, it liberates engineers from repetitive and labor-intensive tasks.
Testing, for example, is an area where Copilot could potentially be helpful for software engineers. Writing tests for software can be a tedious and time-consuming task, so much so that the result is often being unable to run an optimal number of tests, leading to problems later on. By automating these tests, AI augmentation may help engineers examine the integrity of their code more thoroughly than would be possible being done manually.
Deconstructing the Limitations
While Copilot is undeniably a remarkable tool, it is crucial to acknowledge the inherent limitations of AI technology. Here are some general areas of concern we’re going to be mindful of as we move ahead with this project:
- Inappropriate Suggestions: While these tools can propose solutions, there may be instances where the suggestions are incorrect or inappropriate. AI can’t fully comprehend the higher-level logic or business rules that are specific to a project or a company’s needs.
- Contextual Understanding: AI tools may not fully understand the wider context of the code they are working with. For example, they may not grasp the specific purpose of a software module or how it fits into the broader system architecture, which can limit their effectiveness.
- Creativity and Innovation: AI tools can assist with routine tasks and provide suggestions based on existing code, but they are not yet capable of truly creative or innovative coding solutions. They can’t think out of the box or propose revolutionary solutions.
- Data Security and Privacy: AI tools require access to a codebase to function effectively. For sensitive projects or industries where data privacy is paramount, this could be a potential security risk.
- Lack of Human Touch: Lastly, software engineering isn’t just about writing code. It involves understanding user requirements, cooperating with other team members, and making judgment calls – nuances that are beyond the current capabilities of AI.
- Less Effective for Complex Tasks: While even anecdotal reports seem overwhelmingly to suggest Copilot will be helpful with repetitive and boilerplate tasks, can it really increase productivity with custom code and within the constraints of complex business rules?
- Dependency on Training Data: AI augmentation tools generate suggestions based on patterns they have learned from their training data. If the training data is not diverse enough, the AI might fail to offer accurate suggestions for complex, high-level coding scenarios.
Conclusion and Speculations
At Mission, we are genuinely excited to explore AI augmentation programs like Copilot. Our ongoing experimentation, integrating Copilot within our current projects, will yield invaluable insights into its effectiveness and limitations. Despite the challenges that lie ahead, we are curious and excited to see how Copilot might make life easier and results better for our developers.
We’re also already opening up a dialog internally about why or why not this technology can or will replace human developers. Right now, it does feel like the day when clients can use AI alone and skip out on hiring developers altogether is quite a ways away. But as that isn’t the main scope of this project, we won’t say more on the subject here. For now, stay tuned for more updates as we gather more data and real-world use cases of this tool on our active missions.
It’s also worth noting that even in the short time since we began this project, many competitors to CoPilot have since cropped up. We intend to confine our project to Copilot to make collating the data clearer. We’ll certainly update you should we adopt any other tools along the way.
