Tag: Django

Database connection leak while using threads in Django

Oftentimes we need to call a third party API in our web application. The recommended way would be using a task runner and task queue like Celery and RabbitMQ.

But sometimes, in a small project it is overkill to use Celery. Especially for a small traffic web site. On the other hand, I don’t want to call external API directly because it will block the request and user may have to wait, and that is never a good thing, small traffic or not.

So I decided to use different thread to call API. And it worked for my purpose.


from threading import Thread
class APIThread(Thread)
def __init__(self, data):
self.data = data
Thread.__init__(self)
def run(self)
# call external api
# process return value
# create new thread to call api
def make_thread(data)
APIThread(data).start()

view raw

call_api_thread.py

hosted with ❤ by GitHub

Everything was fine until I write the unit tests. When I run the tests, an error was thrown at at the end, saying the test database cannot be dropped because there was another user using it.

Usually Django will automatically close the database connections when they exceeds maximum age defined or are not usable any longer.

There was a database access using Django model object in the new thread. After a quick search on google, I found out that the database connection were not closed even after the threads execution finished. The connections need to be manually closed.


from threading import Thread
from django.db import connections
class APIThread(Thread)
def __init__(self, data):
self.data = data
Thread.__init__(self)
def run(self)
# call external api
# process return value
# Django model database access
# close all connections in the thread
for conn in connections:
conn.close()
# create new thread to call api
def make_thread(data)
APIThread(data).start()

view raw

thread_close_connections.py

hosted with ❤ by GitHub

By placing the above code at the end of the thread function body solved the problem.

On a separate note, if you are using uWSGI in deployment, you need to add

--enable-threads

argument to enable multi threading.

 

 

 

Slow Django-Vagrant-VirtualBox On Windows Host

I’ve been learning Django Web framework for a while and started working on a few projects using it. I use Vagrant with VirtualBox as my development environment on Windows machine.

Issue

It gets slower and slower to run tests and runserver command locally. The reason is VirtualBox’s vboxsf used by Vagrant synced files has performance issue when there are large number of files/directories.

Solution

It seems that it was a known issue and the solution is to use NFS for synced folder. I search in Vagrant documentation , and it said that

Windows users: NFS folders do not work on Windows hosts. Vagrant will ignore your request for NFS synced folders on Windows.

So according to Vagrant’s documentation, NFS folders do not work on Windows hosts.
We need third party plugins to make it work on Windows host. After searching the net for a few hours, I stumbled upon a program called WinNFSdBinary which is a NFS server for Windows.

After setting up the NFS folders with Vagrant, the tests and runserver command run fast again.

Steps

Download WinNFSdBinary which is a NFS server for Winodws.
https://github.com/dziad/WinNFSdBinary

Follow instructions from wiki https://github.com/dziad/WinNFSdBinary/wiki to install and setup NSF in Windows.

In Vagrant file, add the following lines,

Vagrant.configure(“2”) do |config|
  # ....

   # VirtualBox provider needs private network setup for nsf to work
   Config.vm.network “private_network”, type: “dhcp”
   Config.vm.synced_folder “.”, “/vagrant’, type: “nfs”
end