High-performance network requests in Python 3.6

I’m in two social Slack teams at once. UQCS has a thousand members and a thousand custom emoji. The other has a few tens of members and no custom emoji.

I wanted to duplicate some of the finer emoji to add ‘core functionality’ like :this:, the :thonking: family, the :partyparrot: family and a few language/product icons.

Slack-Emojinator exists, and it can both download and upload sets of emoji. I wanted to do some of my own HTML parsing to get experience with BeautifulSoup, so I wrote my own version of the first half.

Now, Slackmoji-Swarm1 is my faster method to bulk-export emoji, while remaining compatible with Emojinator’s bulk upload.

Scalability is important, because 1,000 emoji is actually quite a lot - the naive solution took over 5 minutes to download on my home connection.

Long story short, I wrote three versions: one naive, one reusing connections in requests, and one full rewrite with aiohttp (“asyncio http”). The results speak for themselves.

Progressive improvement in runtime

Now that I’ve written it, tested it, profiled it, and used it, it might be worth submitting as a pull request to the Emojinator project.


It’s been submitted to slack-emojinator! I recommend the project in general as more fully-featured, and particularly once the aiohttp code goes in.


Thanks to:

aiohttp’s documentation

“Illia”’s writeup of high-performance HTTP in Python

Check out the project on GitHub!

  1. Names are hard.