Response headers not being set !

[Axnjr]

I am working on a reverse-proxy server, I want to set the headers that i get after fetching the original server but it does not works. I receive 9 headers from original server but only 4 are added to current headers.

CROW_CATCHALL_ROUTE(Server)([](const crow::request& req, crow::response& res) {
	string ip = get_server_ip(REQUEST_COUNT, IP_POOL);
	auto op = fetch_response_from_ip(ip, req.raw_url, PORT);
	
	if (op) {
		res.code = op->status;
		res.body = op->body;
		if (req.raw_url != "/") {
			for (const auto& header : op->headers) {
				//std::cout << "Header: " << header.first << ": " << header.second << std::endl;
				res.set_header(header.first, header.second);
			}
		}
		// All headers set in the response are being logged correctly but they are not present when checked in the networks tab in dev tools 
		// and file contents are missing. I recive 9 headers from original server but only 4 are added to current headers 😢
		std::cout << "Headers set in response:" << std::endl;
		for (const auto& header : res.headers) {
			std::cout << header.first << ": " << header.second << std::endl;
		}
	}
	else {
		res.code = 502; // Bad Gateway
		res.write("Failed to forward request: Unexpected Error Occurred !");
	}
	res.set_header("BODY", op->body);
	cout << endl << "BODY: === " << endl << op->body << endl;
	res.end();
});

[gittiver]

Could you make a fully reproducible example? I do not really believe that headers get lost that way, but I will check if I have a reproducible example.

[Axnjr]

Sure,

Dependecies:

• crow
• cpp-httplib (https://github.com/yhirose/cpp-httplib/blob/master/httplib.h)

Expected: All headers from the original request should be forwarded to the destination server.

Actual: Some headers appear to be missing when the request is forwarded.

#include "httplib.h"
#include "crow.h"
#include <iostream>
#include <vector>

using namespace std;
using namespace httplib;

int main() {
	crow::SimpleApp Server;
	CROW_CATCHALL_ROUTE(Server)([](const crow::request& req, crow::response& res) {
		httplib::Client PROXY_CLIENT("172.27.48.1", 3000); // port to be added
		auto original_response = PROXY_CLIENT.Get(req);
		if (original_response) {
			res.code = original_response->status;
			res.body = original_response->body;
			if (req.raw_url != "/") {
				cout << "ORIGINAL HEADERS:" << endl;
				for (const auto& header : original_response->headers) { // 
					cout << "(" << header.first << " , " << header.second << ")" << endl;
					res.set_header(header.first, header.second);
				}
			}
			// All headers set in the response are being logged correctly but they are present when checked in the networks tab in dev tools 
			// and file contents are missing. I recive 9 headers from original server but only 4 are added to current headers 😢
			std::cout << "Headers set in response:" << std::endl;
			for (const auto& header : res.headers) {
				std::cout << header.first << ": " << header.second << std::endl;
			}
		}
		else {
			res.code = 502; // Bad Gateway
			res.write("Failed to forward request: Loadbalancer Error");
		}
		res.set_header("BODY", original_response->body.substr(0, 100)); // just to check
		res.end();
		});
	Server.loglevel(crow::LogLevel::Info);
	Server.port(4000).multithreaded().run();
	return 0;
}

[gittiver]

In case of multiple headers with the same Key(name) this works not correct, as res.set_header(...) is implemented as:
/// Set the value of an existing header in the response.
void set_header(std::string key, std::string value)
{
headers.erase(key);
headers.emplace(std::move(key), std::move(value));
}

This erases the previously set header of the same name.

add_header(...) allows to set multiple headers withe the same name/ key, so it should correct the behavior
for multiple header with the same name.

If this solves your problem, please tell us. Then we should only correct the documentation comment and maybe the guides for set_header and add_header.

BR & HTH

[Axnjr]

I have tried that too res.add_header(header.first, header.second);

When I log the crow res.headers they all are indeed visible:

Headers set in response:
Accept-Ranges: bytes
Headers set in response:
Accept-Ranges: bytes
Content-Type: image/jpeg
Cache-Control: public, max-age=0
ETag: Content-Type: W/"2526f-18eff52d3f1"
Connection: application/javascript; charset=UTF-8
Cache-Control: close
Content-Length: public, max-age=31536000, immutable
ETag: W/"ea4-190efe7a1ce"
Connection: close
Vary: Accept-Encoding
Content-Length: 3748152175
Date: Mon, 29 Jul 2024 13:05:07 GMT

But are not actually present when checked in the networks tab, and the files received are empty

[gittiver]

screenshot does not fit to Log output (css vs. jpeg).

[Axnjr]

Here's the corresponding log for the file: fd9d1056-1214565e4835fb55.js
There are too many logs for 17 files, so the wrong one got pasted 😅

(2024-07-30 08:41:20) [INFO    ] Response: 000001A1A7795EB0 /_next/static/chunks/fd9d1056-1214565e4835fb55.js 200 0
IP:Success (no error) (0)

Headers set in response:
Accept-Ranges: bytes
Content-Type: application/javascript; charset=UTF-8
Cache-Control: public, max-age=31536000, immutable
ETag: W/"2ba5-190efe7a1d0"
Connection: close
Vary: Accept-Encoding
Content-Length: 11173
Date: Tue, 30 Jul 2024 08:41:20 GMT
Last-Modified: Fri, 26 Jul 2024 16:37:33 GMT

[gittiver]

Does the raw view look different? What software is it?

Can we have a tcpdump of the response?

[Axnjr]

I get this when i use curl:

When i don't touch headers:

curl -v localhost:4000
*   Trying [::1]:4000...
*   Trying 127.0.0.1:4000...
* Connected to localhost (127.0.0.1) port 4000
> GET / HTTP/1.1
> Host: localhost:4000
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 49251
< Server: Crow/master
< Date: Wed, 31 Jul 2024 02:05:32 GMT
<
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="preload" as="image" href="/sparkle.jpg"/><link rel="stylesheet" href="/_next/static/css/4cd4294a4f81ba5a.css" crossorigin="" data-precedence="next"/><link rel="preload" as="script" fetchPriority="low" href="/_next/static/chunks/webpack-4c4507ba1771c9b2.js" crossorigin=""/><script src="/_next/static/chunks/fd9d1056-1214565e4835fb55.js" async="" crossorigin=""></script><script src="/_next/static/chunks/69-ab04e6ffaa3271be.js" async="" crossorigin=""></script><script src="/_next/static/chunks/main-app-0feae9fd5f66c546.js" async="" crossorigin=""></script><script src="/_next/static/chunks/0e5ce63c-f7058a40ccf3b988.js" async=""></script><script src="/_next/static/chunks/250-b06eddc369cac47a.js" async=""></script><script src="/_next/static/chunks/590-fe53818fdf20f232.js" async=""></script><script src="/_next/static/chunks/291-e038b806cc30d23e.js" async=""></script><script src="/_next/static/chunks/876-67eda2600b536601.js" async=""></script><script src="/_next/static/chunks/416-4266cf6895579a42.js" async=""></script><script src="/_next/static/chunks/app/page-fca3581de8b43222.js" async=""></script><script src="/_next/static/chunks/126-f5cd5579ed369362.js" async=""></script><script src="/_next/static/chunks/app/layout-7720d2f7c116c9d2.js" async=""></script><script src="/_next/static/chunks/app/not-found-c7e86425feb9f575.js" async=""></script><title>Ignition - Ultimate real time platform</title><meta name="description" content="Generated by create next app"/><script src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js" crossorigin="" noModule=""></script></head>
.... long html document 

When i use add_headers method:

curl -v localhost:4000/
*   Trying [::1]:4000...
*   Trying 127.0.0.1:4000...
* Connected to localhost (127.0.0.1) port 4000
> GET / HTTP/1.1
> Host: localhost:4000
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Cache-Control: s-maxage=31536000, stale-while-revalidate
< ETag: "7gbc9d7tnq1200"
* Connection #0 to host localhost left intact

[gittiver]

Any middleware in Crow active?

[Axnjr]

no