Database connection leak while using threads in Django

Oftentimes we need to call a third party API in our web application. The recommended way would be using a task runner and task queue like Celery and RabbitMQ.

But sometimes, in a small project it is overkill to use Celery. Especially for a small traffic web site. On the other hand, I don’t want to call external API directly because it will block the request and user may have to wait, and that is never a good thing, small traffic or not.

So I decided to use different thread to call API. And it worked for my purpose.

Everything was fine until I write the unit tests. When I run the tests, an error was thrown at at the end, saying the test database cannot be dropped because there was another user using it.

Usually Django will automatically close the database connections when they exceeds maximum age defined or are not usable any longer.

There was a database access using Django model object in the new thread. After a quick search on google, I found out that the database connection were not closed even after the threads execution finished. The connections need to be manually closed.

By placing the above code at the end of the thread function body solved the problem.

On a separate note, if you are using uWSGI in deployment, you need to add

--enable-threads

argument to enable multi threading.

 

 

 

Slow Django-Vagrant-VirtualBox On Windows Host

I’ve been learning Django Web framework for a while and started working on a few projects using it. I use Vagrant with VirtualBox as my development environment on Windows machine.

Issue

It gets slower and slower to run tests and runserver command locally. The reason is VirtualBox’s vboxsf used by Vagrant synced files has performance issue when there are large number of files/directories.

Solution

It seems that it was a known issue and the solution is to use NFS for synced folder. I search in Vagrant documentation , and it said that

Windows users: NFS folders do not work on Windows hosts. Vagrant will ignore your request for NFS synced folders on Windows.

So according to Vagrant’s documentation, NFS folders do not work on Windows hosts.
We need third party plugins to make it work on Windows host. After searching the net for a few hours, I stumbled upon a program called WinNFSdBinary which is a NFS server for Windows.

After setting up the NFS folders with Vagrant, the tests and runserver command run fast again.

Steps

Download WinNFSdBinary which is a NFS server for Winodws.
https://github.com/dziad/WinNFSdBinary

Follow instructions from wiki https://github.com/dziad/WinNFSdBinary/wiki to install and setup NSF in Windows.

In Vagrant file, add the following lines,

Vagrant.configure(“2”) do |config|
  # ....

   # VirtualBox provider needs private network setup for nsf to work
   Config.vm.network “private_network”, type: “dhcp”
   Config.vm.synced_folder “.”, “/vagrant’, type: “nfs”
end

Responsive OneMap UI using media query and flex box

In this tutorial, I will show you how to make responsive UI for web page using flex boxes and media query. I will redesign one of the examples available on OneMap API Help Web Site. We will see how to make pages using OneMap JavaScript API responsive. Please visit the OneMap API address search example  to check the original web page.

onemap_add_eg 1) Inspect the source code

By inspecting the source code, I found that the page is designed using HTML Table. It is quite a simple page with only a few sections, each section embedded in  table cell. I am going to use media query and flex boxes to make the page responsive.

2) Put viewport meta tag in the head of the document.

width=device-width” indicates the screen’s width in device independent pixels. “initial-scale=1.0” to indicates one to one relationship between CSS and device independent pixels.

3) Convert the HTML Table structure into separate sections as follow.

.searchbox and .results are put into .container2 so that they could be moved around together when the screen size becomes bigger.

4) Design smallest view first and use media query for different break points.

onemap_min

5) Slide in the results on the screen when the user clicked the search button.

Edit the displayData( ) function as above to slide in the divResults into the view.

6) Slide back and hide the results list when the user clicked on one of the result, and then show the clicked location on the map. 

Add the following code to ZoomTo( ) function to hide the list.

7) Add media query and styles for break point with minimum width 800px.

In a wider screen, container2, which contains search input box and result list, and map area are placed side by side. Result list is put back on screen by setting transform to 0,0.

onemap_800_min

8) Add media query and styles for break point with minimum width 1000px.

After minimum width 1000px is reached, the contents will stop growing, and left and right margins will be added automatically.

onemap_maximum

That is! We have a responsive page using OneMap as base-map. This is just a very simple rework on the existing code and by no means a complete work. But you get the basic idea of how to make your page responsive if you are using onemap api in it.

Using OneMap Address Search in iOS App

In this tutorial, we are going to implement the address search feature using OneMap REST API. If you want to know more about OneMap, please visit to OneMap API Help page.

We will continue our tutorial from my post how to add OneMap to iOS App, in which I’ve shown how to use ArcGIS iOS SDK and adding OneMap base map and setting map extent. Complete the steps in it before attempting this tutorial.

We are going to call the Address Search REST API from OneMap form our iOS app. I’ve tried out the API call and explored the data returned from the service in previous post Exploring OneMap REST API with Swift Playgrounds.

Add Navigation Controller
First, we need to add Navigation controller to our View Controller. Go to Editor > Embed In > Navigation Controller. A Navigation Controller is added to our View Controller.

Embed in navigation

Add UISearchController And UITableViewController
We are going to use the UISearchController to allow users to key in address. The results will be shown in UITableViewController as the user type in the key words. Add UISearchController property and UITableViewController property in the View Controller.

In the viewDidLoad( ) method, add following setup code for UISearchController.

Next, add the necessary delegates to View Controller as follow:

Create a data structure to hold the address data
Create a structure named Place to hold address name, category, X and Y.

In ViewController, declare an array of Places. This array will be our data model to store the search results.

Implement UITableView delegate protocol methods 
There are three protocol methods we need to implement to adopt the UITableView delegate protocol. Add the following methods to ViewController Class.

Implement UISearchResultsUpdating delegate protocol method
UISearchResultsUpdating protocol requires to implement the following method. This allow us to update search results based on the information users enters in the search bar.

Next, we will implement the function to call OneMap REST API to search for the address.

Run the application
If everything goes well, a search bar will be displayed on the navigation controller title place.

searchbar

When the user type in the search keywords in the search bar, getAddresses function is called and a list of addresses started by the keyword is shown in the table view as suggestion.

Auto suggestion

Show the selected one on the map view
We need to implement one more function to show the search result when the user clicks on one of the suggested places. This function is called in tableView:didSelectRowAtIndexPath: delegate method.

showaddress

Above screen is shown when the user selects City Hall from suggested addresses. The project is available on github. Please leave comments if you have any questions and suggestion.

Exploring OneMap REST API in Swift Playgrounds

I had posted how to add OneMap into your iOS application in previous post. In this tutorial we will try one of the OneMap REST APIs in Swift Playgrounds and see the return results. Xcode Playgrounds is perfect for experimenting with new APIs interactively. We are going to use Address Search API from OneMap.

This allows searching of address data for a given search value.  This also allows support of commands such as AND, OR, including word (+), excluding word (-), and exact phrase search using quotation marks. It will return search results with X,Y coordinates of the searched location.  http://www.onemap.sg/api/help/

First create a new Playgrounds file in xcode. Then import XCPlayground module. After that call XCPSetExecutionShouldContinueIndefinitely and set it true so that execution will continue and we can see the return result from asynchronous REST API calls.

Next, we are going to use NSURLSession to make the REST API call to OneMap service. I use the example URL available in their api help page.

After that we pass the url string to dataTaskWithURL function of the session object and implement a completion handler function to tackle the return result from async call.

We can convert the result JSON data using NSJSONSerialization.JSONObjectWithData method and cast it as NSDationary. And then we can write println statement on the elements of  NSDationary result. The values of the result will be shown interactively at the right column of Playground. We can change the parameter values in the url string and see the result changes interactively.

onemapplaygroundWe will use this API call in the next post, adding OneMap Search Function in iOS App.

Using OneMap in iOS map application

I am learning Swift, the new programming language developed by Apple for iOS and OSX application development, and ArcGIS iOS SDK. I am planning  to make a series of tutorials based on what I have learned progressively.

In this tutorial, I am going to show you how to use Singapore’s official map platform, OneMap (www.onemap.sg), in iOS 8 application.

OneMap is an integrated map system for government agencies to deliver location-based services and information. It is a multi-agency collaboration with many government agencies currently participating and contributing information. ~ http://www.onemap.sg/home

To use OneMap in our iOS app, we need to use ArcGIS iOS SDK from Esri.  First download the ArcGIS SDK for iOS from https://developers.arcgis.com/ios/. Follow the guide to install and setup the development environment. ArcGIS has a very comprehensive guide for installing and setting up the development environment.

To use OneMap, first you need to add a UIView to your storyboard. In the Identity Inspector, change the UIView class name to AGCMapView.

MapView

Connect the map view to the view controller by creating an outlet in in ViewController.swift file.

Outlet

Import the ArcGIS framework in view controller’s swift file to use ArcGIS SDK for ios.

In the viewDidLoad method of ViewController.swift file, create a tiled map layer using OneMap map service and add it to the map:

Run the application and you will see the map application displaying onemap.sg base map in the map view.

Initial Run

Right now, our map displays the whole Singapore, but we want it to be focus on a particular area in Singapore when it is loaded. To do this we need to define the initial map extent to zoom in display. We need to pass the envelope geometry to the  zoomToEnvelope:animated: method to zoom to the area we are interested in. 

An envelope is defined by a pair of X-Y coordinates representing the lower-left and upper-right corners of a rectangular bounding-box.

To create an envelope geometry, we need X min, Y min, X max and Y max. How do we find out this values? Pretty easy. We can get the current map’s extent from mapView.visibleAreaEnvelope property. So we just have to zoom in to the area of interest and inspect the current mapView.visibleAreaEnvelope values and use them as the initial extent in viewDidLoad method.

Our View Controller must adapt to the  AGSMapViewLayerDelegate protocol and set itself as the delegate of mapView‘s layerDelegate.

In the viewDidLoad method, add this line to set delegate. See the API reference documentation for the AGSMapViewLayerDelegate protocol to learn more on layerDelegate

Then implement mapViewDidLoad delegate method to add observer for mapView pan and zoom notifications.

Run the application and zoom to the area you want to show on initial start up. As you zoom in to the area, you can see then envelope geometry data printed out in Xcode console. When you are satisfied with the view, copy the X min, Y min, X max and Y max from the console and use them as initial extent.

I choose Nanyang Polytechnic as my starting view and create an envelop geometry from it. Following is the complete code:

Mat on NYP

In this tutorial, we have learned how to add OneMap as based map to the iOS application and setting up initial map extent to the location of interest by using envelop geometry. If you want to know more about OneMap features and services, please visit to their API help page http://www.onemap.sg/api/help/.

Look forward for more tutorials on using OneMap from me and please leave a comment if you find any error or have difficulty following the steps above.

Batch Geocoding Tool For Singapore SVY21 Coordinate System

I have created a tool to geocode large amount Singapore postal codes into X, Y values of SVY21 coordinate system. SVY21 is the Singapore standard coordinate system used by Singapore Land Authority.

You can get the X, Y values of an address by using search function provided in www.onemap.sg. But if you have a few thousands of addresses to geocode, my batch geocoding tool can help you better. You need internet connection to use the tool because the program is the wrapper written in c# to call web service provided by onemap.

Download link:  Batch Geocoder Installer 

Download the installer and unzip the folder. Run setup to install the tool on your computer. You may need .NET runtime to run the program. After installation run the tool from start menu.

batch-geocode

Simply provide an input file containing a column of Singapore postal codes. Give the name and location of the output file where the result (X, Y) values are to be written. Click Run button to start the batch process. It may take a while depending on the number of postal codes you are processing.

Right now, the tool can only accept postal codes as input. In the future, I am planning to extend it to accept whole or partial addresses as input. The source code is hosted on Github https://github.com/zkkmin/BatchGeoCodeOneMap for anyone who is interested to look at it.