GPT-4 Coding Assistant: Real Feedback After 3 Months

Simi included in AI

2023-06-20 659 words 4 minutes

Contents

Bottom Line First

After 3 months of using GPT-4 for coding assistance: useful, but not as magical as the hype.

Productivity boost is roughly 20-30%, not 10x. More precisely: GPT-4 saved me time on “looking up docs” and “writing simple repetitive code,” but complex problems still require thinking through myself.

Real Numbers

Over these 3 months I kept track:

Task Type	Times Used GPT-4	Times Found Useful	Effectiveness
Doc lookup / API usage	89	81	91%
Write simple functions	67	58	87%
Write test cases	45	32	71%
Explain unfamiliar code	38	35	92%
Refactor code	23	12	52%
Debug	19	8	42%
Architecture design	11	2	18%

Conclusion: the more straightforward and answerable a task is, the better GPT-4 performs. The more judgment required, the worse it does.

GPT-4’s Real Strengths

1. Documentation Lookup

        
# Before: Google "pandas merge vs join"
# Now: ask GPT-4 directly

# Question:
# "What's the difference between pandas merge and join? When to use which?"

# GPT-4 answer:
# merge = SQL-style join, needs on key
# join = wrapper around merge, default left join
# Example: df1.merge(df2, on='key') vs df1.join(df2)

This scenario GPT-4 is nearly 100% accurate, and faster than Google.

2. Explaining Code

        
# Throw unfamiliar code at GPT-4
# Ask: "what is this code doing?"

# GPT-4 accurately explains:
# - function intent
# - key variables
# - potential problem spots

For reading other people’s messy code, GPT-4 is more effective than Google.

3. Writing Simple Functions

        
# Task: write a function that counts words in a string
# GPT-4 output:
def word_count(s):
    return len(s.split())

# Correct, usable

Simple tasks like this GPT-4 basically never fails on, and does it fast.

GPT-4’s Real Weaknesses

1. Debugging (Complex Bugs)

        
# Hardest bug I encountered:
# Python multithreaded program, crashes occasionally, ~1% probability
# No error logs at all

# Ask GPT-4: help me analyze possible causes
# GPT-4 gave 10 possibilities, each sounded plausible
# Actual cause: GIL contention + some library's thread safety issue

# GPT-4 didn't have this context, couldn't pinpoint

Problem: GPT-4 can’t give me information I don’t already know. It can only recombine what I provide.

2. Architecture Design

        
        
        
    
# Ask: "I'm building a real-time chat system, what architecture should I use?"
# GPT-4 gave a very standard answer:
# - WebSocket
# - Redis pub/sub
# - Microservices split
# - Database sharding

# But didn't fit my scenario:
# - 100 daily active users
# - 5-person team
# - Budget: $0

# GPT-4 doesn't know my constraints, so the recommendation doesn't apply

3. Hallucinated Code

        
# Ask GPT-4: give me usage examples for Python library xyz
# GPT-4 provided what looked like professional code
# Run: ImportError: No module named xyz

# This library doesn't exist—GPT-4 made it up

Most likely to hallucinate: niche libraries, uncommon APIs, experimental features.

How to Use It Correctly

Don’t ask GPT-4 questions you don’t know the answer to.

❌ Wrong usage:
Ask: help me choose a framework, which should we use?
(GPT-4 doesn't know your team, stack, deadline)

✅ Correct usage:
Ask: in this scenario, what are the respective pros/cons of Redis vs Memcached?
(You have context, GPT-4 provides information, you make the call)

My Actual Workflow

Here’s how I actually use it:

        
# 1. Documentation lookup → GPT-4 (90% scenarios sufficient)
# 2. Simple code → GPT-4 (saves time)
# 3. Complex code → write myself + GPT-4 review
# 4. Debug → analyze myself first, GPT-4 as second opinion
# 5. Architecture → don't ask GPT-4, think it through myself or ask a human

Conclusion

GPT-4 coding assistance: useful, but only if you know how to use it.

Its value is time savings (docs lookup, writing repetitive code), not decision-making help.

Think of GPT-4 as an tireless junior engineer: can execute clear instructions, bad at making judgment calls.

After 3 months, 20-30% productivity boost is real. Not a gimmick, but not a revolution either.