Month: May 2018

Database connection leak while using threads in Django

Oftentimes we need to call a third party API in our web application. The recommended way would be using a task runner and task queue like Celery and RabbitMQ.

But sometimes, in a small project it is overkill to use Celery. Especially for a small traffic web site. On the other hand, I don’t want to call external API directly because it will block the request and user may have to wait, and that is never a good thing, small traffic or not.

So I decided to use different thread to call API. And it worked for my purpose.


from threading import Thread
class APIThread(Thread)
def __init__(self, data):
self.data = data
Thread.__init__(self)
def run(self)
# call external api
# process return value
# create new thread to call api
def make_thread(data)
APIThread(data).start()

view raw

call_api_thread.py

hosted with ❤ by GitHub

Everything was fine until I write the unit tests. When I run the tests, an error was thrown at at the end, saying the test database cannot be dropped because there was another user using it.

Usually Django will automatically close the database connections when they exceeds maximum age defined or are not usable any longer.

There was a database access using Django model object in the new thread. After a quick search on google, I found out that the database connection were not closed even after the threads execution finished. The connections need to be manually closed.


from threading import Thread
from django.db import connections
class APIThread(Thread)
def __init__(self, data):
self.data = data
Thread.__init__(self)
def run(self)
# call external api
# process return value
# Django model database access
# close all connections in the thread
for conn in connections:
conn.close()
# create new thread to call api
def make_thread(data)
APIThread(data).start()

view raw

thread_close_connections.py

hosted with ❤ by GitHub

By placing the above code at the end of the thread function body solved the problem.

On a separate note, if you are using uWSGI in deployment, you need to add

--enable-threads

argument to enable multi threading.