[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP publishing body size limitation #47690

Open
grignards opened this issue Aug 10, 2020 · 4 comments
Open

HTTP publishing body size limitation #47690

grignards opened this issue Aug 10, 2020 · 4 comments

Comments

@grignards
Copy link
Contributor
grignards commented Aug 10, 2020

Hello.
We are using pushpin for real-time updates in the last version of our product Tracim.

During our tests we got unpublished messages when their size is big. After investigation the problem is linked to be HTTP publishing which limits the content's body size. If I'm not wrong, it is this line https://github.com/fanout/pushpin/blob/e1312a54c0db0a4e381583de6b5e8d3708c67204/src/handler/simplehttpserver.cpp#L38
Publishing the same content through 0mq is fine.

Is the HTTP limit a "hard" one, meaning that pushpin is not meant to handle bigger contents? Or could it be made configurable?

@jkarneges
Copy link
Member

I suppose this ought to be made configurable, and the size limits between HTTP and ZeroMQ inputs should be more consistent. However, large messages are not recommended as they can slow down queues. For comparison, Fanout's cloud service has a 64kB size limit.

Thought experiment: if you had to pick a different size limit and couldn't change it later (meaning it ought to suit your current needs and any future ones), what would you choose?

Also, what transport is being used by the subscribers?

@grignards
Copy link
Contributor Author

OK, it was not our primary intent to publish large messages, more a side-effect of our time constraints.
Currently our biggest messages are around 4MB but we'll likely change their structure to not include all data which would make their size drop around 1kB. We are using Server-Sent Events as transport.
In the mean time we can use 0mq although I would have preferred to stick with HTTP publishing to avoid 0mq's persistent connections between our backend and pushpin. If you're willing to accept a PR making the HTTP body size configurable I can have a look at it.
Is the main effect of "slowing down queues" a delay in message publishing?

@jkarneges
Copy link
Member
jkarneges commented Aug 12, 2020

With the HTTP streaming transport (which you are using if you're doing SSE), it is possible to use reliable streaming to set a URL to fetch data from and then publish a hint instead of the actual data. This will cause Pushpin to fetch the data from the URL and send it to the client, in which case the data has no size limit.

However, it might be easier to just increase the publishing size limit. A PR for that would be great.

By "slowing down queues" I mean if you add a broker layer between your publishing code and Pushpin. Not a problem now but something to be aware of as you grow.

@grignards
Copy link
Contributor Author

Thank you for the additional information.
I just created a PR to add configuration for http max body size and max headers size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants