01 April 2011

Large Debian installations in India - Example of setting Call Center using FOSS tools

Raj Mathur and Tirveni has done a complete setup of Call-in/Call-out
center using FOSS tools

Here is the post

---------- Forwarded message ----------
From: Raj Mathur (राज माथुर)
Date: 2011/4/1
Subject: [ilugd] [LONG] Re: Large Debian installations in India


The client is a large call-out business headquartered in NOIDA with call
centres in 5 other cities in India, including New Delhi.  At the time we
started, they had no IT or automation on the call floor at all.

Before you see the words "call centre" and freak out, let me assure you
that this is one of those professional ones -- any telecaller found
calling a DNC number is immediately and publicly terminated.  In fact,
preventing calling of DNC was one of the reasons they wanted to give up
manual calling and switch to an IT solution where call-outs could be

All the implementation decisions were taken by Tirveni and I in
conjunction with the client's technical team.  We decided to go with
Asterisk for the telephony part and Linux desktops with headsets for the

To answer the specific questions Sudhanwa and Nirmalya put up:

On Thu, 3/31/11, Sudhanwa Jogalekar wrote:
> A. On what parameters selection of a particular distro is done.
> Pricing, support, stability etc. etc.

The key element here was stability and availability of packages.  Pure
desktop distributions were ruled out due to their (typical) quick
obsolescence and, to some extent, lack of testing.  This left the
enterprise-grade distributions like RH, CentOS and Debian.  RH and
CentOS don't have the wide variety of packages that Debian offers, so we
decided on Debian.  Of course, our own affinity for Debian may have
played some part in that decision :)

> B. Is the decision taken by some central IT department and
> imposed on all others or is it coming from user requirements across
> the country?

The decision was made at headquarters.  As I said, the organisation is
pretty raw where IT maturity is concerned, and having a strong,
technically sound, experienced CTO at the helm more or less defined the
direction for the whole company.

Of course, business decisions are still made at the operations level, so
the technology strategy has to ensure that it's in sync with and can
service business requirements in what is, after all, a very dynamic

> C.  How the organisation is going to get support? Inhouse? services
> from vendors or consultants? Outsourced activity completely?

L1 and hopefully L2 support will be handled within the organisation.  T
and I have been working on documenting standard procedures, and in the
past 2 months or so most of them have been handed over to the client's
support team, along with some scripts that make life easier for them
(e.g. quickly make new users -- you wouldn't believe the employee
turnover these call centres have!).  We still handle some L2 and most L3
support, and that is likely to be the model going forward too.

Incidentally, anyone with Linux technical competence interested in a
job? ;-)

> D. What is the typical configuration of desktops, servers.

Desktops are commodity 2GB RAM, 3GHz Pentium dual-core machines.  They
seem to be handling plain voice telephony over SIP just fine.

Servers are much larger -- Asterisk needs a load of power to handle 1000
simultaneous users, and we've split up functionality so that the SIP
handling and the PSTN connectivity are done by different servers.
Servers are typically 2x4 core or 4x4 core Xeon class boxes with SAS

> E. What was the timeframe to complete the project?

We started around mid-December (2010), got the servers by mid-Jan and
had one centre live on Asterisk within about 15 days of that.  Planning
out the architecture in advance made a lot of difference to the overall
speed of implementation.

Tirveni did tons of preseed magic on the desktop front, and we now have
a process where you can put a bare machine on the LAN, select "Boot from
network" (PXE boot) and have a working, customised Debian desktop ready
for use in 10 minutes.

> F. What are the most troublesome situations you face during the whole
> exercise. Technical, manpower handling, financial etc.

AFAIR, the most troublesome portions were (a) handling user creation,
(b) changing business requirements and (c) diagnosing and fixing
Asterisk-PSTN issues remotely.

User creation went through multiple phases of streamlining, until now we
have a process by which a support person can login to a desktop, run a
command, feed in the user ID, get it validated against a central
database and have the desktop ready for the user in about 30 seconds.
It's still not perfect, but we're getting there.

As mentioned before, business requirements keep changing, and keeping up
with them is quite a challenge.  This is not due to lack of foresight on
the organisation's part -- just that business needs, TRAI regulations,
security issues, mandatory controls, etc. are so dynamic.

And if you're sitting in NOIDA and trying to manage an Asterisk box
connected to the PSTN 2000KM away, I have one word of advice: don't!
We're learning as we go along, but Asterisk diagnostics are cryptic to
say the least, and telcos are typically reluctant to acknowledge issues
at their end.  You have to provide tons of evidence and sort of rub the
telco support person's nose into it until he actually takes some action
to fix a problem at his end.

On Thursday 31 Mar 2011, Nirmalya Lahiri wrote:
> 1) How the servers are connected?
> 2) Are they connected as nodes of a big cluster or they are placed in
> different location of the country and form a distributed server
> setup?

Currently each centre is more or less an island.  We're working with the
ISP to be able to tie all the centres together so that we can, for
example, login with a SIP client from NOIDA to, say, Chennai for testing
and so forth.

We were lucky to partner with a very competent networking company for
the LAN portion.  The switch/VLAN design they did is also responsible
for the smoothness of the whole operation.

To sum up, it is possible to run call-out (and by extension call-in)
centres using purely FOSS tools and technologies.  The two most
important things you need are:

- A competent team or consultant who understands the technologies and
stumbling blocks involved, and
- Commitment from the organisation's management and technical leaders to
the solution.

Given these, there is no reason why a FOSS solution cannot surpass
proprietary, commercial solutions in features and performance, and
undercut it thoroughly where pricing is concerned.


-- Raj

By- Narendra Sisodiya

No comments: