Investigating request payload sent using tcpdump

Use case: Usually, when integrating with external service endpoints in your backend application, you will be using libraries (require('https') for Nodejs, import requests or import aiohttp for Python, HttpUrlConnection for Java, etc.) to make HTTP calls. It might be difficult to debug what request headers and payload are sent to remote instances; in this case, tcpdump can show how the request is structured (headers and payload) and forwarded to the server; this will help debug.

Make sure you have tcpdump installed. In general, if the service you are integrating is on http, you can use a command like below to see how the request is sent. First, using ifconfig find the network interface used to communicate with the internet. And you can use tcpdump to see the network traffic sent to an external service.

tcpdump -i utun2 -qettttAS -vv host integrated.service.com

Note: here utun2 is the network interface and integrated.service.com is the service endpoint host to which the request is sent.

You should see request headers and payload, but for the HTTPS endpoint, you would see encrypted data. In this case, you can start a simple web server using - python3 -m http.server 4000 (you can use any port here.)

$ python3 -m http.server
Serving HTTP on :: port 4000 (http://[::]:4000/) ...

All localhost/127.0.0.1 traffic goes to another interface, mostly lo for Linux and lo0 for macOS (you can check using ifconfig). You can use tcpdump like below and use port instead of host.

tcpdump -i lo0 -qettttAS -vv port 4000

Now in your application, instead of the actual service endpoint, you can put http://localhost:4000 and make the call; you should be seeing something like below -

Host: localhost:4000
consumer.id: 1111-1111-11111
svc.name: abc-svc
svc.env: stg:1.0.0
sec.key_version: 1
consumer.intimestamp: 1645211463000
sec.auth_signature: aVgPTNGxsrHs9WiddSIgcPH2wl7EPU6eeFIg==
qos.correlation_id: 1234
content-type: application/json
Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: Python/3.8 aiohttp/3.7.4.post0
Content-Length: 436


2022-02-18 11:11:04.276214 IPv4, length 56: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->3cc2)!)
    localhost.terabase > localhost.64452: tcp 0
E..4..@.@.................6..'.......(.....
..a.....
2022-02-18 11:11:04.276309 IPv4, length 492: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 488, bad cksum 0 (->3b0e)!)
    localhost.64452 > localhost.terabase: tcp 436
E.....@.@................'....6............
......a.{"notification": {"data": {"dynamic_data": {"products": "[{\"product_image\": \"https:\\/\\/test.com\\/is\\/image\\/test\\/0040615720019_A\", \"product_name\": \"Gel Pen, Black Ink, Medium - 12 Pens\", \"product_url\": \"https:\\/\\/test.com\\/p\\/1700\"}]"}}, "email_recipients": ["test.name@mail.com"]}}