Home > September 07 Roundtable Part 3

Save

Edit

Cancel

More actions

September 07 Roundtable Part 3

Tags:

Church IT Roundtable at Granger, September 26, 2007, Part 3

Jason

Let’s get back to the discussions. Are there any more wireless questions? Wireless is a beast. If you can outsource it, thumbs up. So wireless is helpful, you can make a strong case for it, people will rely on it. Make sure you do it right the first time. Willow is a great story. For their tech arts conference, they really built up the fact that they were going to use their wireless to interact with the crowd, but it failed miserably. They had 20-some access points spread across the back, but it did not work. They got rid of those devices and went with an elaborate Aruba deal. In their catwalks, they’ve got omni-directional antennas shooting down to a certain number of seats to make sure they are covered. Fun with wireless. Willow is running a 256K bandwidth.

Our nomadics device, we’ve got around 60 current users, this device handles all our public wireless, it’s supposed to do this dynamic natting where even if you had an IP address statically set on your laptop that’s not part of our scope, it will still get you through as it should. I think the Sonic set up is supposed to do that stuff too. Back when we were researching this, we wanted to be able to handle 300 people and this was 3 years ago and these Nomadic guys were the only ones out there. This is Dean’s very expensive device, a network monitoring tool, both a PC and a Lenox kernel sitting underneath here and you can figure out anything you want to do.

Dean

I asked Jason how much he wanted people to see, I brought it so you guys could see it, play around.

I have a quick comment about threshold email modification, my wife does the financials for our church and she tries to make sure everyone has what they need, she put a full page together on sales tax and things like that, printed it out on bright green paper, put it in everyone’s mailbox, then during [Time Stamp00:07:04] the meeting, people were asking every question that had been answered on that paper, so no matter how you notify them, people are not going to read it.

Tell them what you’re going to tell them, then tell them what you’re telling them, and then tell them what you told them. I think that applies.

If I got a bright green piece of paper, I wouldn’t read it. I wouldn’t take it seriously, so it’s gotta look professional. If I can’t scan it and get the info from bullet points, forget it. People aren’t gonna read much.

We took a unique approach to getting people to read the emails, they went out to some restaurants in town and got free gift cards donated to the church and so every now and then, once a week or whatever, they would send out an email that would look like some of the others, but it would say the first to respond gets this $50 gift card. So they started paying more attention to their emails.

I think it is hilarious to put in the communication “if you’re reading this, do this” and it’s funny how many people say “I never saw that!”

AutoResponder! To every IT email! Ha-ha

That whole conversation shows you what we will do to avoid reading something.

Prime example, our church website, we do podcasts and mp3 downloads, I was at the church 9 months before I realized how to do the podcast because there was a cool icon to do the mp3 download but there was no pretty way of getting to podcast, so I sat in the leadership team meeting one day and said, “I don’t even know how to do that and I’m the IT Director.” So that proved to me that communications aren’t being read. I just look for the pretties on the page and move on.

Jason

Let’s talk monitoring. What do we use to monitor? The alerting piece. We’re still trying to figure out, this is our Internet bandwidth [Time Stamp00:11:35] that is being chewed up today.

We’re still trying to figure out the best tool, I talked with these guys, this is a Tony Dye thing but I changed the wording. Tony’s thing was find whatever solutions stinks the least. There’s no golden cow.

Yesterday our core switched wigged out and was pumping as much data up as it could, so it was lovely for a little while.

Is there alerting built into that, like if you hit max.

Jason

I’m sure there is. There are some threshold alerts and stuff.

We use PRPG [?], I’ve mentioned before, it’s a good product. [I can’t hardly hear/understand this speaker]

We’ve done some exporting of the performance monitor stuff, we jumped through a bunch of hoops to get___ to read that data, it’s been successful, wigs out every once in a while.

Part of the issue with SMP is that it takes up space on your network where WMI is happening at the server first. I want to pull information that is server-based using WMI, switches and other network stuff. We are also playing with system center central, it sounds like the cat’s meow, we’re still playing with it.

If you’re not familiar with it, it is basically the small business version of Operations Manager. We use Operations Manager, we went through several months process, strictly server monitoring, we looked at a lot of products in the different price ranges, the biggest thing that sold us on Operations Manager was that we were a Microsoft shop. Also from a cost perspective, we didn’t have to go in and dedicate somebody to manage [Time Stamp00:17:49] the product because the things we were seeing with the other products that would do all the great stuff, but somebody had to go in and do all the stuff. When we first put it in it was a little bit noisy, took a little tuning. Today we are running probably 16,000 different rules against all of our servers and it seems to be doing a good job.

They’ve started building intelligence into it.

I don’t know what charity pricing is. We pay a fee for the server and then if you want base for less monitoring, maybe 100 bucks for us, if you want to do more exchange, it’s more.

Ski is basically the low-end small business version of that of the new Operations Manager, the biggest difference is that it has all the features of monitoring but you don’t get the event log collection.

The event log monitoring, does it help you determine what to look at.

It has reports. So I can say I want to run a report on whatever. There are a couple of demos out there, from a security standpoint, they moved it to realtime into this database.

Jason

Charity pricing $198 if you want the full-blown option. Just hit my blog and Google Microsoft charity pricing.

Another thing I would point out with Ski Ops Manager that’s pretty cool is, we’re trying to move to this now, we know what’s going on with the server, don’t have a clue what’s going on with laptops and desktops. What are people, if you’ve ever seen the error message that says, “hey this application bombed out, if you want to send a report to Microsoft” or whatever, with the Ops Manager and the Ski you can route that to where it goes to a location on your network and you can go in and look at it.

We’ve had people say “we sent that error report [Time Stamp00:23:06] 10 times, didn’t you get it?” They think we get those Microsoft error reports. It was an educational opportunity for us. They naturally think they are communicating with us.

We want to move toward an environment to where, we have certain desktops that bloops twice in one day, we call them before they call us.

Jason

We want to make everything proactive, even to the point where, I guess one of the HP things is bringing the personal back to the personal computer, so a lot of times in our corporate environment we lock down things to try to keep users from doing stupid things to themselves, installing the latest whatever, so I’d like to get it to the point where the machines are open, you can install whatever you want, but when you install it, I know about it. I get an email telling me who installs what.

You have to use caution with that because then you are seen as the morality/functionality/productivity police, so you have to surround that in the right way.

There’s a legitimate need for you to be able to do that but don’t let them convince you that you are the morality police. No, I’m protecting the equipment, we’ve spent thousands of dollars on the equipment, we are protecting.

Jason

Create a culture that is more friendly, where people are looking to IT. So the mindset becomes ‘before I go download whatever I want, I know that they are going to know that I’m doing this, maybe I should talk to IT first.” So the IT is not this inhibitor, we are there to enable, talk to us. Dustin works for a company that has 2000 laptops and everybody is a local administrator.

Dustin

We have so many thousands of users we manage across the country and probably 99% are local administrators, that way we don’t ever have to deal with any issues. We use Ghost or [?] to troubleshoot.

Our philosophy [Time Stamp00:28:43] on the whole deal, we talked about where people don’t have admin access on their machines, it’s part of our culture, 75% of our people travel over half of the time, and we have very few desktops. It comes down to judgment. If somebody is sitting in their hotel room bored out of their mind and they want to download software to play games, we should allow them to do that.

Our policy is that if you install software without contacting IT first and then you call with any problems…

We put a program on each of the laptops and it’s called Smart Protector Pro, inexpensive, $25, www.regsoft.net Email me and I’ll shoot you the charity pricing link. You can set it up so that every time they log into or off of your network, it copies everything they’ve changed. It’s 100% reliable.

Jason

So if we could figure out some way to know what people are doing and come along behind them and help them, you nix the IT police idea. This is part of why people are looking at the Mac, IT doesn’t control the Mac. The problem you have data that’s not being backed up and we’ve had recently a bad story where everything was gone, not backed up. So if we can figure out ways to make the PC more personal and help people, it will ultimately be better.

It also depends on the policy, now that we have more laptops, that makes it more difficult. Is it just a work computer? If it’s a work computer, it’s gonna function that way, how strict do you want to be?

Smartphone is another thing. You have your own cell phone but you want to get corporate email on it, there are issues with that. This person owns the phone, where do you draw the line. It used to be that my work computer was my work computer, at my desk. Then I’d go home and [Time Stamp00:33:57] work on my personal computer. Luckily, this is an advantage of being in a church where we are 10 years behind the culture, we can see how it happens in the enterprise but we have to think of it, how much warm-fuzzy do we need to have for our users so that they can embrace the tool we are giving them and they go out and create another silo of data or another tool because of the restrictions. When I embrace a tool, if I don’t completely buy into it as the approved tool that works for me in my ministry, I’m gonna go over here and use my cell because it’s faster and easier, the catch is all that causes data not getting backed up, etc so I’m helping the user come along with that. We have a change log, they are going to post in a wiki what they’ve changed, dependent upon the user. We have to decide what our organization wants to tolerate, what are the risks of having someone as a local administrator? Does the benefit out-weigh the risks? Black and white.

This might be opening a can of worms here, slap me if it is. One of the things that’s not always felt right at these Roundtables as an outsider, I don’t work in a church, is how tight you guys are on your laptops and desktops and how you put all these group policies into place and you make sure people can’t do certain things. At the end of the day, I would leave these Roundtables and go, “these guys are pastors who are administrating to people who are changing lives and you tell them that you can’t give them access to such and such because it’s a bluetooth device or a USB key, we won’t allow so and so, I just challenge you guys to make sure you don’t forget what your customers are trying to achieve. Your customers are the ministers, the staff, don’t lock it down so much.

How much of it is reactive of compensating for something else? How much of it is us locking things down because of the unknown and maybe we didn’t train a user that if you do this, your whole hard drive is wiped out. How have we not equipped you to succeed, so we said “no you can’t do this” It’s part of our job to know that there are things that can happen and try to protect our people and our equipment.

I constantly fight my staff. Why’d you block that? Why’d you block this? MySpace, Facebook, yes, we’re going to block those. IT people tend to sit at our desk and make group policy and block sites instead of getting up and going to talk to the people to solve the problem. Let them make a bonehead move, a lost document every now and then goes a long way.

The thing is though Dean, we just had a hard drive that probably was erased because somebody didn’t lock the machine, and the amount of hours that you have to spend because somebody didn’t lock their machine.

My response is also often shock when I hear how much you guys lock people down, I’m always trying to challenge because nobody in the church market has managed more computers than we have, and we’ve just not had the problems that you guys are so afraid of having. Here’s the way it usually plays out, number one, you’ve got an engineer so that you are ready to support it without all the resources, you have to build your strategy around the possibility. Then number two, you have a short training session with your staff and say, “Here’s what we are doing. And if you guys load a program on your system, we can’t quickly overcome, we’re gonna be re-writing your hard drive.” So we’ve told them to save all your files to, if you change that, both files are at risk. Then the [Time Stamp00:41:05] first staff member that goes through a problem, teaches the entire staff. So let them make a mistake, it’ll be minor.

But it’s gonna cost you time!

I disagree that it changes the entire staff, they don’t even learn.

I’ve seen the same thing re-occur, even the same thing to the same person. It didn’t teach him and it’s not teaching the others.

That’s where we all get gun-shy.

There are certain sectors in the ministry that are worse than the others and those folks, typically one they screw up, we lock it down and they have to earn it back.

If you are having that problem with that employee, why not lock them down instead of everybody?

I feel your pain. Everybody on our network is local administrator; they can do anything to their machines at any time. When they really screw it up, it’s trash. They don’t get their laptop for 2 days, so they do feel the pain. What happens is, it’ll happen, some accidents, some not. I don’t care what we do for Mitch, he kills himself. And we finally had to tell Mitch, “You have no rights anymore.” “You have to call us if you want to install anything!” But it took two years to get to that point.

Jason

Then you get in the situation too where it’s Mac integrating into the PC world. So this is brought to light though, some people are missing. We’ve got these Macs that are saving stuff, who knows where, not backed up, whose responsibility is it? so a couple weeks ago I brought everyone together that’s involved in saving lots of storage, typically audio video people, typically a lot of Mac users, we just went through this pain, this rippled through the church. So we’re trying to attack it and say instead of having these individual silos of data, what does a large centralize data-storage do, your data goes here, it has to go here. There will be a checks and balances to make sure your stuff is getting there, [Time Stamp00:45:10] so it is one place, centrally backed up, whatever it is, it has to be in this place. Since we just went through that, there was a lot more head-nodding than there would have been in the past.

We don’t even attempt to back up PCs.

But we’ve got a large base of Mac users that their idea of transferring files is to unplug their [?}] drive and drag it down the hall. And that’s where the files live until it dies.

We had on our editing system a raid zero array [?] fail that had 3 months of recorded weekend service, probably 2 terrabytes of video data wiped out.

Where would you back up 2 terrabytes of data to right now Jason?

Jason

My solution at the moment is at the raid one. That’s at least my minimum. That’s in our budget discussion right now. We have to find a solution for our media, we’re looking at Fiber [?], huge price tag.

We are in your situation. I went to our media director and said absolutely not. I can at least control that environment.

Whether you’re managing the systems or the people? Know what I mean? Implementing policy so that they don’t get in the way of themselves. We’re protecting the equipment. It’s a blurry line. It has to do with what your user-base is like.

I think part of the reason you look at it from the perspective not being in the church, not being IT Director of a church, there is a significant difference in our hiring practices. And do we as the church solve that problem? Do I have the expectation that is somebody comes in knowing how to use a computer, and if they don’t, have I trained them to do it?

Do you only lock it down for those who need protecting from themselves?

I’m not saying my solutions great, I’m saying the catch is that we’ve done our users a disservice and our reaction is to lock things down. [Time Stamp00:49:01] Do I need to go to my HR Director and say, “We don’t hire anybody unless they can pass a test.” I don’t know.

That’s ignoring our company’s user-base. We are managing over 50 networks and when people say “who do we need as an IT manager?” We can set the bar fairy low because we are going to be doing a lot of the policy setting for them so they don’t have to work through those issues. That’s just who we are. The point is that the users are your users and we are not experiencing the concerns that you guys are guarding against. I don’t think you have a user base that comes near what we deal with.

I don’t know what your support model is, when someone comes in and they’ve only used Windows once or something and they are trying to get on the computer, they come to the first human person. We’d like to solve it at the hiring level but at the same time they see a gifted person but they can’t operate a computer.

There are varying levels of users. Volunteers, lock it down. Once they are hired staff, you have an opportunity to influence them and say you can’t install software. It takes weeks months sometimes. I scare the daylights out of my new people. I say, “don’t you dare install it because if you do we have a tool that show us that you just installed software, I know you think it’s free but if you read the license, it probably says it’s free for personal use but this is a corporate environment, don’t do it.” Sometimes they do it, they can’t use the excuse that they didn’t know.

We use Express Metric for our auditing software and it’s simply a piece of software you put in your login script and it kicks off, every morning when you come in, you get reports on what’s changed and what’s loaded.

We have a similar thing. Do you have issues with laptop users? Ours seems like [Time Stamp00:53:03] never get inventoried.

[Several people speaking at once]

Every one of ours is connected to the domain, they have to run these certain things before they can do anything.

We have ours so that the laptops have to go to Jim once a month for an hour, scheduled by them, recurring appointment, they are responsible to bring it to us, we review it physically and check everything out. They have to do that. We have it back to them in an hour. We’ve also gone a different route, they have an administrator password so that when they are traveling and they need to add something, they can. How can be at the point where we are not yet allowing them to be a local admin but how can we facilitate them to do what they need to do.

We were just talking about content filtering and when you think about, you have no compunction about content-filtering the staff, and yet, it’s time to lock their computer down so they can’t install software, when it’s a big thing to give somebody admin to install software and not give somebody else those rights. Where does stewardship begin and end here? I don’t understand.

The issue is culture. We’ve got to process culture. I work more on culture than I do on group policy, content-filtering, instant message blogs, anything else. I find that probably once a year, I go through about 10 days where I do extensive research on who is abusing our system. I don’t content-filter at all. These people are adults and I’m going to treat them like adults. Now if they do something that’s going to cause damage to the machine or they download something that’s a known virus, I’m quickly visiting them, but looking at somebody’s surfing logs and things like that, I just don’t do it. During those 10 days, I find stuff [Time Stamp00:56:53] I wish I didn’t know. I don’t work in a church so I have more exposure to the secular world, so I see things but I deal with it. In IT I hold a higher standard, I’ve terminated two people in the last year for breaking policies where anybody else would have gotten a slap on the wrist. I deal with people swiftly when they break policy when I know about it. So what I tell people on that first day is, “Look, this computer is yours, we give it to you to use for company purposes and if you break these policies, we will find out and we will show up at your desk.

Once you give somebody admin rights though, aren’t you endangering them to software exploits?

Sp

Potentially, sure.

At least you are limiting the front door.

Well we actually aren’t as strict as a lot of places, I think it’s interesting that we talk about this because all of our laptop users are admins, desktop users here we lock down except for a certain piece of software and we don’t do content-filtering on our network except for the worst stuff that Sonic wall blocks. For us, we’re not the IT nazis. If they are offsite, they need to be able to do what they need to do. If they bring it back and it’s hosed, then they’ve got no computer for 48 hours or whatever.

Part of that is organization budgeting.

Honestly that’s going to be one of the last items approved in normal budgets.

It saves, but it’s that upfront cost.

When you push back and you have to cut something out of the budget, that’s gonna be one of the first things to go. When we budget for workstations, we tack that cost on as part of the budget. It’s not negotiable for us.

It’s only $12 per seat for [Time Stamp01:01:14] Ghost Enterprise Suite.

We pay $14.

CCB has one of the best source for shrink-wrapped software.

Dell will always beat CCB’s pricing. And CCB won’t negotiate. Dell has hands-down beat CCB’s pricing.

With Ghost though, you’ve got to keep that Ghost image up to date. We have Ghost too, there are updates that Microsoft has released or whatever.

That’s a non-issue if you’re running a [?] server, you bring your image back and let it sit on the domain.

There are other updates and issues.

Acrobat Reader

What it comes down to is none of this stuff is easy, whether you use Ghost and keep it updated or just start from scratch every time like me cause all I’m putting back on it is OS and Office.

But you take 10 minutes to get the image back on there, then how much time is it going to take to get things updated?

It’s a good time to keep your images updated.

I guess I’m going to ask a question that probably no one is going to have an answer for but I’d like to get your opinions. As a vendor from the outside looking in, why is it the culture in church IT, because it’s not just this room that we’re locking things down at ridiculous levels.

I think I’m a moderate when it comes to that scope, I’m definitely not where Jason is, I’ll call myself a moderate.

Is it because as a church organization, there are no teeth [?] in the policy, and that’s at an HR level so you’re not going to get canned if you break the rules, is that where it comes from? Or is it because we as IT Directors are afraid of what we don’t know. Why do we do? I’ve never worked in an environment outside the church, so doing church IT is all I’ve done. To hear you say, from the outside looking in, that’s odd. What is it with our breed of IT that causes us to react this way?

I [Time Stamp01:05:33] don’t have an answer but I have an opinion. I have a very varied background. I’ve been inside churches all my life, my father is a minister, grew up in church. I understand the culture. I preached almost every Sunday for 32 years. I have an advanced degree in Theology. Part of that training exposed me to counseling with other ministers. I doubt the 51% figure of people in ministry hitting porn site is accurate, it’s low. I know that stat is higher. What we’re struggling with in the IT are all those issues as well as are we administering or service [?] and we’re trusting to some degree the ministry staff and the leadership to say we know what ministry is to our target audience, this is what we need to do and this is the best approach that we believe to reach those people. And as IT people, and everybody that’s tied all together in this, we need to support that direction for that particular ministry, whatever that means. Rather than trying to make our jobs easier so we don’t have to work too hard, which I think at times I get the feeling that part of IT is really not trying to help us but trying to make it so they don’t have to work any harder than they want to. We need to be supportive of the ministry. But the lines are blurred, where is it we start helping the process. For example, Covenant Eyes, my guess is that a large percent of large churches when they get to staff and try to implement that would be voted down. And the reason is who’s voting. The abuser. So [Time Stamp01:08:22] it’s going to be hard to implement the very tool we need that would help them the most. To me that’s troubling. I try not to think about it very much. How far do we go? What we really get bounced back to is how to clean up the messes that are created.

Jason

That’s got to do with resources.

Everybody is understaffed.

I also spent a few years in retail. JCPenny store manager and what I found in my second year there was we had to cut a number of employees, my third year, same thing, my tenth year, same thing. There’s always more to do. That’s an excuse.

Do you think part of it is that the role model that we have of corporate America the way that IT is run in those arenas, the stuff you read about, all the security threats?

Sp

The thing that is the challenge is that we recognize that the problems exist, and we recognize that we play a very vital role with support of the ministry and at the same time provide them counseling and service as to the best practices and the best reason to do that, helping with budget stuff, giving them all the information we can to help them make those decisions, and at the same time be available when we can. It’s a problem we live with because that’s our job.

I think one of the reasons why this local admin thing is so big is coming back to a source issue with regards to the operating system, probably all of us are dealing with Microsoft Windows. This is not to bash Microsoft, this is just the way I see it, they’ve got an operating system that is full of holes that they don’t even know about all of them, so what they’ve had to do is teach us how to use their system in a way that minimizes the impact of those holes. So what they’ve taught us is don’t let users log in as local admins. Now we have clients [Time Stamp01:11:53] all the time who will say, “Hey we got this new guy who is going to be our IT administrator and he’d like to get certified in Microsoft or whatever other products” and should we give him the resources to do that? Our response is always that’s it’s good for them to get a broader perspective, we would just ask them to agree in writing one thing, they will try [someone coughed, I couldn’t hear what he said]

We already know from having so many networks what works and what doesn’t. We don’t mind being challenged but don’t change anything until we’ve talked about it. With the admin issue we’ve been taught how to make up for our weakness in the operating system. If you engineer a little bit differently how your networks are set up, it frees your users up to accomplish more.

Jason

Let’s break.

0 Comments Show recent to old