Buying knowledge?
By Rod Deluhery
A slight problem
The other day our
dispatch center called our computer office, saying “the network is slow”. For a large city police dispatch center, this
is not a good thing. It wasn’t a busy
day, but the dispatch people where still a bit annoyed at the problem. Jim, the dispatch super gave me the weird
look.
“You know, this is the same thing that was happening last
time, and it was the network. It wasn’t
a CAD problem”, Jim said.
“True, but I don’t see anything wrong. . .yet”. I said, a little worried but no panic yet.
I continued to look at the problem, thinking it was probably
the server or database. Finally I
started doing some more network tests.
“I think I found part of the problem. . .we are losing pings
on the network. It’s network related, at
least part of it, anyway.” I said.
Suddenly the problem was mine and my blood pressure went up.
I scrambled to find the problem. Using Ethereal, I captured some data on the
CAD network vlan, nothing there. Then I
captured data on the other ports of the switch.
Whoa, massive amounts of data flooding the network. Aha, this must be what is slowing it
down! But what was it, all this UDP
traffic? Soon the deluge of data
stopped, but dispatch was not quite satisfied.
“What can we do to prevent this next time?” the dispatch operator asked me.
“Well, we can discuss it, after I find out what it was. But there are some things we can do. One is to buy a dedicated network switch for
CAD users, and separate the CAD network.“
“How can we better isolate the problem next time? Jim asks, trying to find a resolution.
“Well, I can write up some things on how to better
troubleshoot these issues. Well I
suppose I could document some things to test next time you see this.” I say,
feeling a little better that I actually know something about the problem
now. What happened that made this
incident different than any other computer problem? The issue was it brought a metropolitan 911
dispatch center to a point where they could almost not dispatch calls. A bad thing!
Post mortem
People who understand technology realize there are many
parts that make up our complex machines and systems we use. Look at a helicopter, for example and compare
it to a large IT system. Both have many
interconnected, single parts that, if they fail, the system can be crippled,
unable to perform the mission at hand.
System engineers, technicians, and the operators that use the equipment
all have various levels of knowledge on how IT systems work. The question is, what level, or what training
can give users and operators enough information to help diagnose problems?
As a general rule, ignoring some theoretical discussion,
technical training can be defined as:
Risk = complexity divided
by training. As a formula, R =C/T, meaning the more training, the less
risk. And the more complexity, the more
exposed risk. Some engineers will argue
that they can build in complexity and redundancy that minimizes risk. I’d argue that. The general tendency on many systems is to
build IT that only has a “red warning light” type signal that works similar to
a car warning engine light. Instead of
having five different gauges, the output is summed to one red light. From operational point of view, this makes it
easier for operators or technicians to be alerted that something is wrong,
meaning they have less complexity, less objects to keep track of. There is a downside though. This makes it difficult to identify what
exactly is causing the problem. As a
manager of technology, you need to keep aware of dumbing-down your staff into
just “red light watchers”.
The trend to simplify complex technology is going to
continue. In my humble opinion, such
simplification does not mean you can skimp on training. Operational expenditures (opex) will climb if
your staff does not understand how to troubleshoot complex systems. Saving money on theory of operation, and
diagnosis will end up costing you in downtime.
The take away lesson
from this: Make sure your staff can
diagnose those “red lights” quickly and effectively. Examine the downtime costs versus training
costs to minimize service outages.
The invisible and silent bullet.
Here is a question.
What is invisible to the human eye, silent, can penetrate a building,
cause computer systems to crash, and blow fuses? No, it’s not a virus. The answer is radio waves! We all know that radio waves, or RF (radio
frequency) emissions can be dangerous.
It’s common knowledge that the 800mhz signals from our cell phones
penetrate our brains and can cause cancer.
But we all live with that risk, and play the game of life taking the
risk along with all the other chance exposures, like crossing the street and
hoping not to be run over by a truck.
What is interesting to me is how we all have come to live with these
invisible waves, surges and swells of electrons that pass through us
everyday. What to worry, they are part
of modern civilization and it’s a cost of doing business, correct?
One problem with the ignorance of exactly WHAT radio signals
surround us, is that we don’t have a baseline, and we ignore what is
normal. Similar to the adage of if you
don’t measure something, you can’t improve it.
Except here, the improvement we are talking about is to understand radio,
so that you can prevent interference which degrades performance. Or to understand radio so you can identify
who sent the radio signal that triggered the bomb. So what should we know about RF?
In my opinion, large organizations that rely on multiple
radios systems must have some radio engineer on staff. He may not need to be an EE with a degree in
RF design, but at minimum a radio tech or staff person, who can use a spectrum
analyzer and is familiar with the radio signals your organization uses.
The take away lesson
from this: Ensure that there is
someone who knows at least a baseline of how your radio systems work, from a
radio frequency standpoint. That means
they need to know how far their signals should travel, how often they get
interference, and so on. For large
organizations, an RF expert who can diagnose interference, sources of radio
emissions, and perform baseline checks of radio equipment is mandatory. Ability to use a spectrum analyzer (and have
one on hand) is important.
Survey of police and sheriff radio.
A survey was done of 152 small and
rural police and sheriff agencies in all 50 states. The survey asked the organizations what
technology they had and what they didn’t have.
Respondents were asked to indicate
the extent to which different types of voice communications technology are
available to their agencies. Responses were as follows:
All Some Personnel
Mobile radios 6.5% 3.5%
Mobile phones 54.1% 38.8%
WiFi 7.5%
11.3%
Satellite phones ----
9.6%
Voice over IP 2.4%
6.1%
I come from a large police agency, where every officer has
a mobile radio (walkie-talkie), but looking at these numbers for small county
agencies shows that many DO NOT have mobile radios. It is an interesting case for more services
over commercial mobile phones, where a convergence of technology is ready. Convergence of web services, trunked radio,
and other rich media services can be delivered over existing commercial
cellular phones. Modern cellular phones
can provide map services, web services, connections to other voice systems, and
any other internal network system.
Blackberry phones are an interesting example of
convergence. With a blackberry enabled
phone, all web communications start INSIDE the corporate network. That means internal resources and websites
are AUTOMATICALLY available to blackberry web users, via the internet
blackberry server. The blackberry server
relays all data from the provider radio network into the internal network. A very valuable asset!
At the city I work at, blackberry’s are used to browse the
inside phone web directory page, ping devices inside the network, and many
other unique functions. At this shop they also use a computer aided dispatch
program called Versaterm. Computer aided
dispatch is usually called “CAD”, and the vendor is now gearing up to have a
small version of the program that will work as a Java applet on
Blackberrys! Having some CAD
functionality on a blackberry device is very handy for the officer on the
street who walks or bicycles.
The take away
lesson from this: Information is being delivered into many new
devices and that give the public safety person more options for mobility. When deciding the purchase of pagers, cell
phone technology or other devices, keep in mind what future functionality
people will be requesting.
Pandemic planning, and working from home.
Disaster recovery planning is
being challenged by the possibility of avian bird flu or other outbreak that
would leave workers stuck at home for weeks at a time. The worst case scenario is that avian bird
flu mutates into a fast spreading communicable disease. The challenge for public safety, county and
city services is this. How do you still
provide services when your staff can not leave home or is simply afraid to
enter into the normal workplace?
City jobs that require physical
labor will be severely impacted as workers call in sick. But the computer operations that many people
do all day long can be done from home. Remote
computing can assist in some situations.
The challenge is how to make it work securely and easily from users
homes. Remote computing can help when
they are either afraid to come to work or have symptoms of the disease
themselves. Remote computing, or
telecommuting, has been around for years and is used by many corporations. How can such technology be used?
For a brief example of how remote
computing would help an organization, we will start with a critical city
service, paying the bills! Verifying pay
records, verifying overtime and assimilating all the businesses and services
who need to get paid is a huge job. The
final product after the batch runs is to create the paychecks for distribution.
The city users who do this at their
computer would need to be able to do this from their home computer in a way
that is secure and gives the users access to the same resources they have in
the office workplace. What are the
challenges to doing this? The first is
to find out what are the most important tasks that keep the city running, like
check writing. Parcel out the tasks into
high and low priority, in order of how often and the amount of people it takes
to do the task.
One the tasks are broken into
priority and number of users, those applications that people use can be put on
remote access servers in such a way that most home based users could run the
application from their home pc. That
means minimal amount of software installed on the users home computer, and a
secure connection to the city resources either via internet, dial-up or
radio.
This would allow the users to run
these applications from the computers at home without endangering themselves or
others.
The technology exists to solve the
problems discussed here, the question is how much money are governments willing
to spend? Are they ready to spend money
on building systems that are only used for emergencies? There has been more than one local government
that recently stopped paying for offsite disaster recovery services, even in
the light of Hurricane Katrina. Setting
aside funds for disaster recovery systems is like setting aside money for
anything that is not critical. The money
is competing for many other hands and people are fighting for those funds.
Let’s go over some of the steps
needed for pandemic planning, and disaster recovery.
- Ensure
you incorporating lessons learned!
Every past disaster has lessons that should be used in revised
plans
- Incorporating
human impacts into the recovery and planning process
- Prepare
for the human nature aspect, to prepare staff for the psychological
effects
- Educate
and communicate with employees to increase their awareness
- Leveraging
executive and management buy-ins
- Testing,
and enhancing work from home procedures and options
- Merging
private and public sector best practices to enhance emergency preparedness
- Enhancing
communication techniques during disasters and measure the effects
- Building
business unit, and departmental preparedness to cross train employees
In closing, we state the obvious,
that fighting for funding may be the biggest challenge in making a disaster
recovery system come to exist.
Operational expenses do exist when trying to duplicate any service, and
like anything that is not used every day, it needs to be tested now and then. This all adds up to operational expense.
The take away lesson from this: Disaster
recovery solution implementation has to come from the top, where the money is
distributed and reallocated. Initial
expenditures and operational costs must be taken into account and put in the
budget for a system to work reliably year after year.
Authentication
= Smart cards.
History shows us many examples of civilized people creating
complete chaos because of lack of authentication. Forged letters, redirected orders, and faked
treasures are woven into the timeline of civilization. How do you ensure authentication?
One way to help ensure a computer user is actually who they
say they are, is to use two factor authentication. Actually, this applies to someone using any
device, whether it is a phone, a military tank, or anything else. Having two factors means you need two things.
The two things could be a smart card, and a password. Or it could be specific computer and a PIN
code, or the duel keys and procedures used to launch a nuclear missile. Any two things. . .thereby increasing the
things you need to authenticate yourself.
Why do we need to have authentication? Is this going to be another thing to break
and cause problems?
For one, documents are going digital. Authentication is valuable in my opinion
because we need to validate who someone is that last did something. We need to know who last created a criminal
report, or that officer Bob actually wrote up the murder report.
Maybe you have heard that we all need to start encrypting
our documents, which is true. But how do
you KNOW that a certain person encrypted those documents? Strong authentication helps prove that user X
did a certain action.
More agencies, corporations, and public organizations are
making the slow (and sometimes painful) move to smart card authentication.
What are some of the benefits of smart card authentication
for law enforcement types?
- Decreased
chance of password theft from spyware, and loaning passwords.
- More secure
systems, if properly implemented!
- System
integrity. That includes less
chance of mistakes by system administrators and “super users”.
- Users
may actually have less problems remembering passwords, as the random one
time passwords generated by the keyfobs (smartcards) may replace passwords
they normally had to remember.
The take away lesson
from this: Smart cards are just
another way of accessing resources, usually not increasing or decreasing the
effort required to authenticate.
Enforcing smart card use will increase the security of your computer
systems.
The value of powerful lighting is shown here in a house fire
in Calimesa, California.
Rescue vehicle has a boom light that illuminates the entire scene,
extending 15 feet above the vehicle.
This combined with small telescoping lights from another truck fill in
the shadows with light. The result is a
well lit fire scene, with less accidents and faster operations. Operations in
the dark are no fun! Photo copyright
2006 Rod Deluhery.
“A PICTURE IS WORTH A THOUSAND WORDS”
Product focus: DataPath Portable satellite systems
If instant data communication for a disaster, nothing is
more reliable than satellite providers.
Satellites provide an infrastructure that will almost always be available
even when every other local infrastructure system is damaged. Duluth,
Georgia based
DataPath Inc. supplies quick solutions on a trailer mounted system called the
DataPath 3000. Providing IP-based
network connections, network data connections to the internet are easy.
For a remote area, the DataPath ET2000 weighs much less and
is broken into several cases for transport and can but put together in an
hour. National or international
organizations that deal with emergencies, crisis management of any type should
look at how these satellite systems can fit into the operations of the
organization. Ease of use and simplicity
are important for people needing basic phone and internet service. More advanced users and large groups will
need to look at customizing your specific equipment configurations, and then
testing those devices and configs out in the field.
and allow communications.
Technology focus:
Mesh radio.
By the Radio Man
Mesh radio systems is a highly acclaimed technology for
moving data over a mesh of radio devices.
Mesh networks are nothing new, but with the promise of wireless, mesh
pledges to save you money on access points and to extend your range of any such
radio devices. Mesh radio systems are
made up of a broad randge of digital
communication systems. Data networking
in the 802.11 space include products like Mesh Networks (now Motorola) and
Tropos. All these promise high speed, and
a highly reliable connection.
First, I think the most value you get from Mesh is its
reliability. Think of mesh radio as a
way of having multiple paths to get data back to a point. Mesh routing technology looks at costs of
various routes along the mesh and decides the best path. That is the good
part.
No the bad part. My
question on any mesh product is how much latency is created as the data goes
from hop to hop. Mesh is a great thing and can extend high speed RF networks
much farther than normally possible. In my opinion, mesh networks will get
faster and better as the technology matures and more people use them.
As you are all aware, most radio is half duplex, so the radio can only be
either transmitting or receiving, but not both. This half duplex nature
of radio creates small amounts of latency as data is "piped" through
each leg of the mesh. Each radio node may "wait" a few
milliseconds (or longer) for the radio channel to be clear to transmit.
The more hops, the more wait (and degradation) in performance and higher
latency.
Software can only help so much when there are physical limits with the amount
of data that can be moved at one time. Software and protocols can keep
the load balanced and attempt quality of service
only to the limits of hardware.
The take away lesson from this: You need to understand how mesh can affect
throughput, as you may get fast response time when going through one or two
hops. Past two hops, speed and latency may be affected.
Facts
on chemical spills.
I personally learned a few years ago the importance the
emergency community puts on chemicals in the environment. I had found some barrels of what looked like
dry cleaning fluid that seemed to have been dumped in a field, behind my
fathers house. I took matters into my
own hands, calling the local fire department, where a guy told me in a calm
voice said “that they would send someone to look at it, when they had some time”
and didn’t seem like it was something they would do that day. I figured they may or may not ever come out
here. I ended up going to a restaurant
about a hour later, and then about two hours later found three fire trucks at
my front door. The captain walked up to
my car.
“Sir, you can’t just call in a hazardous material release
and then leave. We have been waiting
here quite a while for you.”, he said.
“I wasn’t sure you would even come.” I replied, shocked they
where now really concerned, and had been waiting for me for 45 minutes! And over the phone they didn’t seem to
interested.
“Please show us where the chemicals are.”, the captain said.
As the hazmat team came out to examine the abandoned waste,
I walked back glad that they did respond to the hazardous waste. I later learned they have the same response
whether it’s one gallon of chemicals left in the street of 55 gallons in a
abandoned field. And it makes sense to
do so. With that, let’s look at some of
the procedures for chemical spills.
For large spills, the federal government request you report
them to the National
Response Center. The phone number is 1-800-424-8802. For large spills, the federal government may
possibly re-imburse the local agency.
Emergency Planning and Community Right-to-Know Act (EPCRA).
http://www.epa.gov/superfund/programs/er/index.htm
THOUGHT PROVOKING, MUST BE IN YOUR BOOKMARKS, LINKS!
http://www.project25.org/
"Project 25 (P25) is the standard for interoperable
digital two-way wireless communications products and
systems."
http://www.apcointl.org/frequency/
From there I found another group called afc/apco. This site has some good information on 800
mhz, 700mhz and rebanding. It is the non
profit group that does frequency coordination.
http://apcointl.org/frequency/documents/Joint_Letter-APCO-IACP-
NSA-MCC-MCSA-IAFC_Final_011306.pdf
Here a letter, actually many opinions to the fcc regarding
800mhz rebanding issues.
Recycled bits, lost and now found. Random data found on the web.
WebEOC is a software product, accessible through the
internet that allows City staff to
track key elements of an event that is accessible to all
individuals on the response team.
WHEN to initiate a
WebEOC Incident? A WebEOC incident or event can be created regardless of the
nature of the event. If it is useful to track event
information, accessible to anyone on the team through the
internet, then use the product. However, by agreement of
public safety managers, all events that meet the
following thresholds will be entered and managed through
the use of WebEOC.
• Lasting longer than four hours;
• Involving multi-departments; and
• Disrupts the City (e.g. traffic, congestion, noise, smoke,
evacuations, etc.)
or
• Whenever a DOC or the EOC is activated.
WHO should direct the
initiation/creation and “population” of a WebEOC incident? This is the
accountability of the incident commander or department head
who is leading an incident that meets the
above thresholds. Simply direct a person who has access to
the internet to follow the “how” steps set forth
below.
HOW to create a WebEOC
incident is easy. Just follow these steps (which include notifying the
organization that an incident has been created). Don’t
worry about creating multiple incidents or creating
ones that aren’t used because the incident was resolved. We
can “clean” things up later. What’s
important is to make use of the product to aid in the
smooth operation of the City’s response.
1. Create the incident by going to the internet and
entering the following:
External: https://weeoc.anaheim.net
a. Access WebEOC through the following steps:
b. Enter Additional Login Information as requested (your
name is all that’s required)
c. At the WebEOC screen:
i. Click Admin
ii. Click Incidents
iii. Click Add Incident box (right middle)
iv. Enter a name for the new incident
v. Save
d. Logout
2. Send an ENS that says:
a. Subject: WebEOC Incident Activated--_____________
b. “A WebEOC incident has been created. The incident name
is _________. The
__________ department is now populating the initial fields.
Responding departments and
personnel should:
i. Go to the internet
WebEOC Guidelines
d. Focus first on getting information into the
“Jurisdiction Situation Report;” then the
“Communications Plan;” with a running dialogue of how the
event is progressing being
entered on the “Significant Events” screen.
e. These are the screens you need to populate and keep
current (in addition to “Sign In”):
i. Jurisdiction Situation Report
ii. Communications Plan
iii. Significant Events
iv. Initial Damage Assessment
v. Mission/Tasks
vi. Maps
vii. Organization Chart
viii. Press Release
ix. Shelter Status
x. Jurisdiction Notification Status
xi. Reports: Significant Event Chrono and Significant
Events
xii. Links: EVOC, Insider, etc.
Additional thoughts:
• You may want to send hourly ENS messages similar to that
described above to let the team know
the incident is being tracked through WebEOC.
• Remember the additional tools you have as a manager in Anaheim to assist in
responding:
o To
hold audio meetings: Conference call phone bridges and “pink” common channel
o To
share digital information: Microsoft LiveMeeting
o To
organize a response: DOC’s, EOC and field command posts
o To
answer phone calls: 7th floor phone bank (eight phones in Conference Rm #2)
o To
see what’s happening: EVOC (Enterprise
Virtual Operations
Center) on-line
End of random bits.
Things that helped
make this issue: Green tea, pistachios
by the cup, Anritsu Spectrum Analyzers, Google Toolbar, Microsoft Word, PDF995
free pdf converter, HP Notebooks (with that kick-ass EtherTronics 802.11
antennas built in), grant money from
UASI, taxpayers of Anaheim, maple tea from Canada, Textpad, Namo Webeditor, and
an occasional nod from my dogs.
Ending quote
"The wireless telegraph is not difficult to understand.
The ordinary telegraph is like a very long cat. You pull the tail in New York, and it meows in Los Angeles. The wireless is the same, only
without the cat. “-- Albert Einstein
Issue
#3 
