hakk

Understanding HTTP from Scratch with Python Sockets

Most Python developers reach for requests when working with HTTP — and for good reason. It’s convenient, safe, and hides all the messy details of networking. But hiding those details also hides how the web actually works.

Under the surface, every HTTP transaction is just plain text sent over a TCP socket. Understanding what happens at that level gives you insight into how browsers, APIs, and servers communicate. It also helps you debug connection issues, craft custom clients, or even build HTTP servers yourself.

In this guide, we’ll peel back the abstraction and send HTTP requests the hard way — by hand. You’ll see how to:

This is a reference-style deep dive, intended for Python programmers who understand the language but want to learn what really happens between client and server.

Sending a Raw HTTP GET Request

An HTTP request is just text sent over a TCP connection. The first line defines the method, path, and protocol version (for example, GET / HTTP/1.1). It’s followed by headers, and an empty line (\r\n\r\n) marks the end of the header section.

import socket

# Create socket and connect to server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('example.com', 80))

# Send HTTP GET request with required Host header
s.sendall(b'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n')

# Receive and display response
print(s.recv(4096).decode())
s.close()

Takeaways

Changing the path (for example, GET /about HTTP/1.1) retrieves a different resource. Forgetting the Host header or using the wrong line endings often results in a 400 Bad Request.

Parsing the Response Headers

The server’s response mirrors the request structure: a status line, a list of headers, an empty line, and then the body. To extract and process only the headers:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('example.com', 80))
s.sendall(b'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n')

response = s.recv(4096)
s.close()

# Split headers from body
header_data = response.split(b'\r\n\r\n', 1)[0]
headers_text = header_data.decode()

# Parse into dictionary
headers = {}
for line in headers_text.split('\r\n')[1:]:  # Skip status line
    if ': ' in line:
        k, v = line.split(': ', 1)
        headers[k] = v

print("Status Line:", headers_text.split('\r\n')[0])
for k, v in headers.items():
    print(f"{k}: {v}")

What’s Happening

Handling Edge Cases in Raw Parsing

Real-world HTTP responses aren’t always well-behaved. A few situations can trip up simplistic parsing logic:

Always validate the first line of the response (HTTP/ prefix and token count) before trusting it, and use higher-level modules for production code.

Checking Response Status Codes

HTTP status codes indicate whether a request succeeded, failed, or redirected. To extract and classify the code:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('example.com', 80))
s.sendall(b'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n')

response = s.recv(4096)
s.close()

status_line = response.split(b'\r\n', 1)[0].decode()
status_code = int(status_line.split()[1])

if 200 <= status_code < 300:
    print(f"Success: HTTP {status_code}")
elif 300 <= status_code < 400:
    print(f"Redirect: HTTP {status_code}")
elif 400 <= status_code < 500:
    print(f"Client Error: HTTP {status_code}")
elif 500 <= status_code < 600:
    print(f"Server Error: HTTP {status_code}")
else:
    print(f"Unexpected status: {status_code}")

Status codes fall into standard ranges:

Validate that the response begins with HTTP/ before attempting to parse numeric codes.

Requesting Specific Paths

To request something other than the root path, modify the request line to include your desired endpoint:

import socket

host = 'example.com'
path = '/api/data'

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, 80))
s.sendall(f'GET {path} HTTP/1.1\r\nHost: {host}\r\n\r\n'.encode())

response = s.recv(4096)
s.close()

print(response.decode())

Always include a leading / in the path. For query parameters, append them as part of the string (e.g., /search?q=python+sockets&limit=10). The socket connection itself doesn’t know about paths — that information exists purely in the HTTP layer.

Using http.client for Production

Raw sockets are educational but fragile. Production code should use http.client, which handles headers, status codes, chunked encoding, and HTTPS automatically.

import http.client
import urllib.parse

def fetch_url(url):
    parsed = urllib.parse.urlparse(url)
    if not parsed.netloc:
        raise ValueError("Invalid URL: missing hostname")

    conn = http.client.HTTPSConnection(parsed.netloc) if parsed.scheme == 'https' \
           else http.client.HTTPConnection(parsed.netloc)

    try:
        path = parsed.path or '/'
        if parsed.query:
            path += f'?{parsed.query}'

        conn.request('GET', path, headers={
            'User-Agent': 'Python http.client Example',
            'Accept': 'application/json'
        })

        resp = conn.getresponse()
        headers = dict(resp.getheaders())
        body = resp.read().decode()

        print(f"HTTP {resp.status} {resp.reason}")
        return {
            'status': resp.status,
            'reason': resp.reason,
            'headers': headers,
            'body': body,
            'is_success': 200 <= resp.status < 300
        }

    finally:
        conn.close()

result = fetch_url('http://example.com')

Why Use http.client

For most applications, requests builds on top of this foundation and adds convenience features like sessions, cookies, and retry logic.

Summary and Next Steps

Manually crafting HTTP requests teaches what really happens when you call requests.get() or open a URL in your browser. Each step — connecting a socket, writing headers, and reading raw text — corresponds to a layer of abstraction that high-level libraries simplify.

Key takeaways:

  1. HTTP messages are plain text sent over TCP.
  2. The request and response are structured with a start line, headers, and optional body.
  3. \r\n\r\n separates headers from body.
  4. Status codes determine how to handle responses.
  5. Production clients use http.client or requests for safety and convenience.

If you want to deepen your understanding, try:

Understanding HTTP at this level bridges the gap between network fundamentals and application-level development — giving you better intuition for debugging, security, and performance.