Tuesday, September 06, 2016

Red Hat's Jen Krieger on DevOps feedback loops

The focus on metrics and feedback loops in software development tends to be around technical measurements like uptime. Or, if we're being really clever, the business outcomes associated with those numbers. In this episode, Red Hat Chief Agile Architect Jen Krieger argues that the human side of feedback loops may be the most important.

Show notes:

MP3 audio (14:55)
OGG audio (14:55)


Gordon:  Hi, everyone. I'm here with Jen Krieger, the Chief Agile Architect with Red Hat again. In the last podcast, we were talking about distributed teams and in general how to make distributed teams work effectively and presumably leading to get better outcomes in terms of delivering software, because that's the name of the game after all.
I'd like to follow up with Jen with something else that we were discussing came out of the Agile 2016 Conference. That was the idea of feedback loops. I've been talking alone with my colleague William Henry at some events about this idea of metrics. How do you know if your DevOps is working?
The focus tends to be measuring the throughput per second or the latency or the number of successful deploys or the down time. Those are important as are the business outcomes that stem from them.
For all the talk we have about the culture in DevOps though, we don't talk an awful lot about how you monitor how the team is doing, both from a day to day basis and in a longer term. What are some of your thoughts on that, Jen?
Jen:  Yes, it is absolutely true that when we think about the words DevOps, and we think about feedback loops, we immediately go to the space in which we're thinking about some sort of computer somewhere giving us some sort of data that will help us make a decision about some sort of thing that we're stumbling over, but we rarely, if ever, think about the fact that every single thing that we do with other people is a feedback loop, having visual confirmation of your idea.
Say you're in a meeting and you're sitting with a bunch of people. You are saying, "What do you think about my idea?" Somebody sitting across the table leans in and maybe crosses their arms and raises an eyebrow.
You might take this as a visual form of feedback and not really know what to do with it. There are all these other methods of feedback that we're getting on a daily basis that we have zero idea of what to do with.
I recently wrote an article for opensource.com where I was talking about feedback loops. I was talking about the idea that for everybody in the DevOps space ‑‑ and this might be a very salacious thing ‑‑ you just need to stop thinking about the feedback loops that you're worried about at work.
All the computer feedback loops, and all those things that come out of a machine, you just need to stop worrying about them right now. You need to start focusing on the human feedback that you're getting and the human feedback loops that you're trying to cultivate, because, I guarantee you, those feedback loops are about 90 times harder to resolve than whatever's going on with your computers.
If your production server is down, don't ignore that. I assure you, you can resolve [laughs] that problem.
If you are having a fight or a negative interaction with somebody that you're working with, I guarantee you that, if you cannot figure out how to deal with that feedback loop, you will continue to not be able to deal with that feedback loop, and it probably will also impact other relationships that you have around that particular interaction.
One of the things that I've been kind of thinking about lately that might help folks processing feedback is to also recognize that we live in a digital age where there is so much feedback coming in that sometimes it's hard to identify what feedback we actually need to listen to.
We were talking last podcast about distributed teams, and the nature of IRC, and the amount of information that comes in across just one form of communication. If you are a person who likes to participate in open source communities, email will be a problem. Trying to keep up with email from a popular open source project like kubernetes is impossible.
It's just impossible. You can't do it. So at some point you have to start identifying what feedback is the right feedback to focus on. That is going to also be critical to your addressing the entire loop.
Gordon:  What you say there in terms of having all these numbers, metrics, what have you, the fact is it's true from a technical standpoint, too. I was at the DevOpsDays and I think it was London earlier in the year, and we were talking about metrics. We were having an OpenSpace about metrics. What do you really want to measure?
One of the participants, he made the comment that his company, they used to get this monthly report that tracked, I think it was 2,000 something metrics associated with their IT systems, and nobody ever looked at this thing.
Even if we're just talking about the technical aspect of things, much less the entire system, it includes people. It's really important to think about what matters. Maybe you won't log all that other stuff, and use Splunk, and maybe do predictive analytics, but it's not something you should be focusing your attention on, on a day‑to‑day basis.
Jen:  The win here is not the number of data points that you're logging. The win is, "If you're logging the data, is your company actually using the data that you are logging?" Because, frankly, logging data costs you money, and it costs you a lot of money. It's not for free. If you are using Splunk, you pay them based on the amount of data that you're storing.
Yeah, [laughs] just to produce a report that no one is doing anything with is not the ideal feedback loop.
Gordon:  It may have been the same event. Somebody from ‑‑ I think it was Google ‑‑ made the comment that data you don't use has a negative ROI.
Jen:  A couple years ago we were talking about that. It was something like black data or dark data, or something like that. [laughs] It was ridiculous buzzword. In any case, the most critical part about feedback loops is not just receiving the data, it's understanding what to do with it, which is what I was alluding to before.
You can be getting a bunch of information coming in, so data from somewhere is the first step. You're actually getting some sort of feedback from something, whether it be human or machine. Identifying which data you want to respond to is important. Figuring out what it means when something happens is important.
Then, somehow, emotionally tying value, and that's an individual thing. The example I used in my blog article was the fact that my husband...he's a video gamer, and he plays a game called "Dark Souls." He continues to run around in this game and die over and over and over again.
We were having a simple conversation in which he told me that he had a lot of souls, which is the monetary system in the game. I said to him, "If you die, you're going to lose all your money."
He's like, "Oh, no, no, no. That won't happen." I was like, "Sure," but I knew for sure that he was going to die again. Sure enough, a day later I heard him cursing in the basement about this. I said, "Sure enough." He died and he lost all his souls.
I looked at that situation and I thought, "Well, gosh. I'm thinking about feedback loops, and I should assign that whole thought process that I've been having on feedback loops and figure out how I might have compelled him to change his behavior in order to not be in the situation right now where he's angry in the basement."
I thought, "Well, instead of me just saying, 'you're gonna die,' I could have said something to the tune of, 'You've got a lot of money, and there's all these things that you can buy. You could upgrade your character. You could do all these things with it. What are the things that you want, that you haven't done yet? You don't have to spend it all.'"
I know he's frugal, which is probably why he's not spending it. I thought, "You can spend half of it, and just go ahead. It's OK." He could have actually spent half of it and gotten a brand new sword or something and been happier for it, but he didn't.
It's the lack of assigning some sort of emotional value to it, or emotional feeling to it that prevents people from understanding feedback, or really participating in a loop.
If you go all the way back to that early conversation about stand up and where it becomes just a status meeting for everybody, it's because no one is assigning, or at least buying into, the idea that it’s actually going to improve the situation for the team. No one really understands how it does that. No one really sees the value of participating in the activity, and, therefore, they don't.
That is the fundamental part in the feedback loop when things kind of go south. You can monitor the heck out of your computer systems, but, if you fundamentally don't care whether they're up or down, it doesn't matter. It just doesn't. That's when you can...sure, monitoring what data you should be doing on your systems, throughput, all that good stuff is really critical.
"What is the most valuable data you should be capturing?" means absolutely nothing if you don't care about the data. Caring about the data is a human quality. You can't make somebody care about it, they just have to want to.
Gordon:  Talking about teams and for the help of teams, what are some of the things as Agile Architect, Agile Coach that you really think about, that you really keep your eye on?
Jen:  The engagement of teams. I tend to pop in and out of a lot of team meetings. I keep my eye on a lot of different people. We, as an industry, have an engagement problem. We have a bunch of people who are incredibly highly paid thought workers who are probably taken advantage of a little bit.
I don't mean that in a way that we should have higher egos, but we are also ‑‑ especially in the United States ‑‑ in a situation where we'll just work ourselves to death, and we're just going to keep working. Not a lot of people pay a whole lot of attention to the boundaries that one would hope to expect from work.
Some of the simple things I look for from teams is to see whether they're going to fall down is whether or not they're aggressively answering email over the weekend. Whether or not I get into a meeting with them and they immediately dive into a heated argument or a stilted conversation,
Or do they spend maybe the first couple minutes just chatting about whatever. It could be chatting about something that is completely unrelated to what we came to talk about, or they could be excitedly talking about some technology. That's fine, too, but they're having an interaction where most of the people on the phone are actually participating. I'm basically looking for engagement. How engaged are people?
For video conferencing, are people smiling? Do they look like they're going to fall out of their chairs because they're so tired? I'm looking for that kind of stuff. Also, too, a lot of my job is just making friendships with people so that I can notice when things aren't going well.
It's not always like a science, and that's part of the problem in this, because the monitoring of computers is really easy. You can set parameters and universally use them. Regardless of what company you work at or what server you're using, it's pretty much universal, but humans are so different.
If I've got one engineer who I know is never going to smile because it's just his personality, I can tell when he's angry about things because I know him, and I know for other things to look at, but if I got an engineer that I know is smiling all the time and suddenly she's not smiling anymore, then I know there's something going on there and I can try to dig in a little bit and see if I can help her with something. It's really just about making human connection.
Gordon:  To close out this podcast, what advice would you give to our listeners to use feedback effectively in the context of people and teams?
Jen:  One of the most important things is do not participate in feedback loops just because somebody told you to. Hands down, the worst thing that you could possibly do. If you are being asked to do, maybe, your annual performance review, chances are I'm not going to tell you to stop participating in that.
That's probably something you need to do, but the other thing that you probably should consider is that, if you are already in a space where you're not going to listen to something that your boss or the person doing your performance review is going say to you, chances are you're not going to get any value from that at all.
There has to be some sort of connection you make to the information that you're getting. If you're not really participating in a, say, daily stand up because your team is already talking nonstop in other communication streams during the day, maybe you want to talk about, "Do we actually need to sync? Do we need to sync about work?"
Maybe it's not work that we have to sync about during that 15 minutes. Maybe it's just to get on the phone and say, "Good morning," to everybody, or see people's faces for a few minutes a day. Maybe it's not an enforced thing that you do just because you're told to.

Honestly, just don't do it if it's not providing value for you, but, on the other hand, make sure you understand the value that it's supposed to provide before you make any sudden decisions about it.

Wednesday, August 31, 2016

Gordon's Hafftime Issue #10

This issue talks:
Gartner Catalyst
Big Kubernetes clusters
Docker goings on

Become a subscriber.

Podcasting redux redux


After something of a hiatus, my Cloudy Chat podcast is once again publishing. (iTunes feed. It’s also now on Google Play and available directly from my blog.) My regular co-host had moved onto new responsibilities at Red Hat and with a lot of travel and other things going on, I just let podcasting lapse. But I’ve got a new episode out with Red Hat’s Jen Krieger talking distributed teams, another one mostly in the can, and am partway through refreshing some of the standard audio and graphics associated with the podcast. I also expect to experiment with the format a bit going forward so you’ll likely see (well, hear) some approaches that differ from my standard 15-30 minute straight interview.

With this relaunch, I’ve gotten some questions from colleagues asking how I go about recording and editing podcasts. I’ve written on this topic before, but reading back, I see that I’ve made a fair number of changes to how I do things. So time for an update.

Let me note at this point that I use a number of different approaches depending upon the circumstances. 

I’ve written about how I record an interview with a remote guest previously and that description still pretty much applies except that I now generally use BlueJeans rather than Google+. But it really doesn’t matter; the process should be pretty much the same for most video conferencing systems. 

If I’m on the road, I try to minimize gear and use one of the iRig microphones plugged into my iPhone. If I recorded this way a lot, I might invest in a dedicated recorder. Note though that some of the high quality ones don’t work well as hand-held recorders because they’re overly sensitive to being handled while recording. (The TASCAM that I describe later on has this property; you have to mount it and/or use external microphones.)

For this post, I’m going to describe how I record interviews when I’m in the office and don’t mind bringing in a (small) bag of recording gear. So this describes a case when your co-host(s) or guests are often in the same location as you are. I’ve fiddled with my kit over the past couple years and I’ve settled on this setup as one that is pretty straightforward once you have it down, can give excellent audio quality, and works to create a natural-feeling interview environment.

My hardware is as follows:

Setup the recorder for the external mics. I also use auto levels to try to balance the volume of the microphones but it doesn’t work as well as I would like. But, with my configuration, the mics record on different stereo channels so I can do some manipulation before I blend them. (I also have a USB sound mixing console that I can use to attach more microphones and to directly control their volumes but I’ve found, for most purposes, this adds a lot of complexity without a lot of benefit.)

For editing software, I use Audacity which is Free and Open Source and has far more features than I need or use. It also runs on Linux, Macs, and Windows.

Once I’ve blended the channels, the first thing I usually do when editing is go to Noise Removal under Effect, get a noise profile, and then apply noise removal to the entire recording. I think this capability was introduced in version 2 and, in my experience, does a great job of removing ventilation and other consistent background noises. Of course, the quieter an environment you can find, the better.

After editing in Audacity, I prepare the XML file that’s needed for iTunes and Google Play, splice my standard intro and outro audio onto the beginning and ends of the podcast, and upload the XML and the audio files to AWS. Most of the podcast apps get their feeds from the iTunes API these days, so that’s the one upload that really matters. Even with Google Play, you need to go through the steps to get your podcast into their store, but after that it can draw from the same XML file that iTunes uses. The only separate upload I do is to Soundcloud.

If it sounds kinda fiddly, it is. I’ve written a Python script [1] to take care of the splicing, the XML creation, and the AWS uploading. (Basically, I fill out a form with the title and description, point it at the MP3 file, and it does the rest. The only manual steps I still have to do are uploading to Soundcloud and writing a blog post for the episode; I also typically get my episodes transcribed by CastingWords for inclusion in the post.)

[1] It occurs to me that this is probably a good use case for Amazon Lambda but I wrote the original version of the script a few years ago and it generally works fine. 

Tuesday, August 30, 2016

Podcast: Distributed teams with Jen Krieger

Increasingly, software teams aren't all working in the same office. Or maybe some are and some aren't. Red Hat's Chief Agile Architect Jen Krieger discusses her experience and offers advice for how to make distributed teams work effectively from appropriate tools, to practices, to behaviors.

MP3 audio (18.21)
OGG audio (18.21)


Gordon Haff:  Thanks for joining us today. I'm Gordon Haff, technology evangelist with Red Hat. Today, I'm joined by Jen Krieger, who's the Chief Agile Architect at Red Hat. We're going to start off this session talking about these distributed teams.
We're not going to talk about whether teams should be distributed or not and rehash all the debate about individual productivity versus team productivity and so forth. Because frankly, our experience at Red Hat is that distributed teams are the reality. Not every software engineer in the planet can live in Silicon Valley, or indeed wants to live at Silicon Valley.
Jen, introduce yourself please.
Jen Krieger:  Hi. Jen Krieger, as Gordon mentioned, Chief Agile Architect at Red Hat. Basically, what that means on a daily basis for me is I just roam the halls here at Red Hat and hope people understand what that word, Agile, means.
I'm talking about distributed teams. It's an interesting topic here at Red Hat mostly because we actually have been distributed since day one, work with our open‑source communities and we can't expect everybody to be in the same room all the time. We're very familiar of this idea of having a distributed working environment.
One of the things that I think about a lot when I'm thinking about the word "Distributed" is, going back to that whole concept of what forms of communication are what we call high bandwidth or the most valuable forms of communication. A lot of people in the Agile space would say something to the tune of, "Face‑to‑face communication is the best way to have a conversation with each other."
Actually, today, Gordon and I are sitting across a table from each other where normally we would get over some form of distributed video conferencing to do a podcast. We're actually uniquely in the same room, which is not standard for us at Red Hat.
What I like to tell people is generally, when you're talking about face‑to‑face communication, video conferencing, email, IRC, all these different methods that you can use to communicate, it doesn't mean that face‑to‑face is the only way to communicate nor does it mean it's the perfect way to communicate.
Some people actually prefer not to have a face‑to‑face communication because there might be language barriers, personality barriers. Maybe it's easier for them to think about what they want to say before they actually are meant to say it.
There are other concerns and other considerations, especially in the world of software engineering today. it's not that it's bad to be distributed, we just have to recognize as an industry what it means. It's the same with face to face communication. That can't be your "Mode one" of communication, if you would. You have to think about the impact that other forms of communication may have on the ability for you to communicate the ideas that you're having.
Gordon:  We're going to get to talking about some of the tools and techniques for using those tools and some of the best practices, but one thing I'd like to start off with is we had talked about distributed teams. We talked about non‑distributed teams. The reality is that there's usually going to be some sort of mix of those things, and that mixture can be challenging itself.
Jen:  In my experience, I've had several different situations. One of the Agile teams I've had most recently, they were all in the same office. There were six of us, all the cubicles centralized together. We had the glass pulled out between them, so we could have that truly collaborative, truly just face‑to‑face type of environment.
It actually worked really well except there were some challenges, because we didn't necessarily want to be in everybody's face all the time. If you are an engineer, it's really sometimes hard to be in a situation where you're constantly pulled out of what you're trying to think about. We would actually pick a day every week where everybody would be working from home, so that folks could focus on just getting their work done.
I’ve also had situations where I would say 90 percent of all the people on the team are distributed, so they're working in their home offices, or one of them is in one of our main offices. There's not really a challenge of trying to share communication, because everybody has to take that second step to communicate. Everybody is distributed. Fundamentally, no one is in the same location.
The third challenge, or the third type of team that I've worked with, is the team where a couple of people are distributed but the central portion of that team is located in a main office and located very close to one another.
They all have their different types of challenges and different ways to address them, but I would say the one situation that is harder to deal with is that third situation, where I'd say maybe half your team is all in one place and half of the team is distributed, so some people are getting the benefit of that face‑to‑face interaction and the hallway conversation, and half the team is not.
Gordon:  Let's talk about that third situation, because I think that's very common, maybe it's even the most common today, in the open source world, very distributed teams are often a norm.
That's maybe not quite a norm at a lot of other organizations but I've certainly seen that myself, whether we're talking software development or something else where decisions get made informally around the water cooler or over pizzas, over beer, whatever, and two or three guys or gals who are in London, in Hawaii, in wherever they're working from, they find out after the fact, "That's right, we've made that decision over lunch last week. Sorry. We'll try and remember to tell you next time."
What are some of the best practices that you've found for dealing with that kind of environment specifically?
Jen:  The best situation that I can recall in my own career is, I would say it's about 12, 13 years ago now. I was the sole distributed team member of a team trying Scrum for the first time. Everybody else was in our Miami office and they were all, on a daily basis, participating in live and active vocal conversation.
I was their Scrum master, sitting in my remote office in Raleigh, North Carolina, trying to somehow manage impediments and manage the team from a very remote place. There was no technology at the time, at least at that office that we could rely on because the company didn't have Slack. It was nothing back then.
Maybe we could've signed up for WebEx but it was too expensive. We didn't really have anything other than a phone. Get that. I was a remote team member, being a Scrum master, and all I had was the phone. A lot of times, what would happen is they would have team meetings and they would forget to call me, because the phone exchange didn't allow direct call in, if they didn't call me, I just didn't go to the meetings.
I became very familiar with the angst of being that remote team member and came up with a long list of things. It wasn't that long. it was top three things that I needed that team to do, which is not forget that I was up there.
We played fun games, we printed out a picture of me. We put it on the conference machine so they would bring it around the meetings so they will remember that I was to be called on the phone. Simple things like that. Now that we have all this technology out there, there are a lot of other things that e can do to integrate that team into a central office.
One of the best things I've seen a team do is have one of those gigantic monitors in the center of a team area and actually having a running live feed of folks to actually join in video conference. Whenever they are thinking to say hi to people, they can just dial in and their face shows up on the monitor.
It's a little awkward if you're in a large office where lots of people walk through all the time. If you are in a small situation where you can actually dial in, you can see people sitting around. It's fun to see that.
The other thing that is really important is, those water cooler conversations are not going to stop happening. They are always going to happen. That's human nature to have a conversation with somebody. If you are sitting right next to them, to not have the conversation is a little weird. You're going to wind up having a conversation.
The things to understand though is that when you are making decisions, the first thing I always like to tell people is, "Who you actually bring to the decision making room is actually a pretty important topic." If you are going to randomly make decisions in the hallway without all the people who need to be in that decision, it's probably not going to be a very good way for you to actually get consensus or drive change to your organization.
If you're going to make a pretty impactful decision, you probably should make sure that you have all the decision makers in the room when you're doing that. Yes, it does slow the decision making down, but in some cases it probably is a good idea.
Example, you're making an architectural decision about your product line. Something needs to change. You're going to switch one technology in for another technology. Is it appropriate for you to have that decision made over a water cooler or should you bring in the extended team who is going to be impacted by this decision to get their feedback on that.
Option number two is probably better than option number one, but it also doesn't mean that every single decision has to be in that central decision making body. I would also say that there are probably some decisions that aren't going to really matter like, "What time do you have your stand‑up meeting?” If you guys want to make a decision and simply say something to the tune of, "Five out of six team members think we want to change it to this time, is it OK with you sixth team member? Would you mind if we do that?"
Trying to delay that conversation until everybody is on the phone may not make sense. It's a matter of determining the decision that need to be made or the question you're asking and how critical is the question, how much impact is it going to have? What's the risk level of the decision you make? Is it really risky? If so, you probably want to talk to more people to reduce the risk of what you might be deciding to do.
Gordon:  Maybe related to that, one of the things that I've often found challenging in my career when I've been dealing with folks who are maybe not right next to me, maybe just in another building in the same location, there are a lot of decisions and a lot of useful conversations just to flesh out thinking, that maybe don't fall into the formal decision making bucket, but they're just part of the day to day interaction.
I find a lot of time, when people are remote from each other, it's harder to have those conversations because you feel like, "It's not really worth bothering that person with a phone call to talk about this half‑formed thought I have." What are some ideas you have about dealing with that kind of thing?
Jen:  It's interesting because a lot of Red Hat doesn't have...maybe they do, but my observation is, a lot of the engineers I work with don't fall into bad habits because they generally use IRC to communicate.
I watch this every morning, when I get into work, people say, "Good morning." Some folks chat about what they did the previous evening. There is a connection that people make during the day. They have general conversation during the day that helps with that cross‑communication.
The interesting part about that is to layer that team dynamic on there where you've got most of the folks located to one office and a few people distributed, I would imagine that a lot of the conversation and the general chit chat happens around cubicles, happens around whatever space that team occupies and rarely, if ever makes it back to IRC, or whatever communication channel you choose to use, whether it be email or maybe Skype, whatever you're using.
It's important for teams to understand, especially if you're committed to doing something together as a team, that you have a social responsibility to the folks that you're actually interacting with. It's not just, get on your team meeting and immediately start talking about work but it's asking somebody about their day, or finding out what they did over the weekend, or trying to make some sort of social connection so that it's easier for you to have casual conversation.
In your own case, you might feel like you can't reach out to somebody and share that half‑baked idea because you don't know them too well. If it was somebody that you knew, you wouldn't hesitate to pick up the phone and have that conversation with them.
Gordon:  When was an industry analyst for a number of years, one of the things I found was that I can certainly connect with people over the phone and have good personal connection with them, but it was much easier if I had actually met that person in real life and had dinner with them, had some beers with them or whatnot.
This comes back to, there is some value in these high‑bandwidth, in‑person communications because often, once you've established those things face to face, that's probably an argument. For example, having a commitment to things like team off‑sites, if you are possibly in a position to afford them.
Jen:  Companies probably make out a little bit by having distributed work force, especially if they're not paying for co‑working space or Internet or a phone, they're putting that burden on their employees, they probably are making a little bit of money by not having to pay for office space or maybe not making money but having it reduced overhead for having to pay for something...a gigantic office building in downtown Raleigh.
The critical point is that it's not as easy to just say, "If I am not paying for a desk, that means I have X dollars free to do whatever else with that I need to," because there actually is the bottom line impact to the way that people are able to work together. Often, because it's not a quantifiable one, as in I can't establish that if I don't get Team A in the same room at least once a year, I can say for sure that they'll have a decrease 10 percent of production. I can't say things like that.
I can't quantify that to a finance person and say, "If you give me $5,000 so that they can all travel to a team bonding exercise or whatever you want to call it, I can ensure that they will have a steady line production for the rest of the year." I can't say stuff like that.
Anecdotal evidence indicates to me that it actually is a critical and important part of everything that we do as an industry. One of the things that I have had happen recently is that I did work with a team that was very distributed. They were from all over the world. They were not really brand new together.
They were having a little bit of trouble actually working well together and coming up with just a general backlog. Everybody was new to whatever it was going on on that space. We brought everybody together so that they can meet each other, have food together. People bond while eating. It's just a thing. It's a universal thing, whether it be beer in the Czech Republic or Italian food in Little Italy in New York City, people bond over eating and drinking. It's just a thing that we do as humans.
By giving them the opportunity to do that bonding, they just worked better together afterwards. That face‑to‑face interaction, it's critical for me, especially being a distributed employee to have that bond with people on a regular basis.
Gordon:  I still remember a few years ago. It must have been a LinuxCon, it was one of the Linux Foundation Events. There was a kernel panel talking about how the development process worked with Linux, kernel and the like.
One of the kernel developers, one of the core maintainers, I don't remember which one it was, but he made a comment about one of the reasons that Linux kernel development has historically worked as well as it has, is that fortunately a lot of people involved...certainly, there's a lot of individual contributors as well but a lot of the core people involved... work for pretty well resourced companies, IBM, Red Hat, Intel, Hewlett Packard, and so forth.
That provides the opportunity and the budget, frankly, that people can get together in Linux Plumbers events, LinuxCons and the like. From their experience, this has been a very important part of Linux kernel development historically even though on a day to day basis, Linux kernel developers are all over the world.
Jen:  They cite the social interaction they're having as their ability to long‑term work well together.
Gordon:  Absolutely.
Let's switch gears a little bit and talk a little more about some of the benefits you see with specific tools. You raised a great point about IRC and Slack...call them messaging but in a sense, they're almost ambient distributed conversation.
To add to your point, when I had this feeling in my product management days that I had to go around and have in‑person conversations because things weren't important enough to bother some of them with an email or with a telephone call. One thing just those kind of tools provide is they do allow for this asynchronous, non‑interrupting type of conversation to take place.
Jen:  It's interesting too because a lot of people think that they need to read everything that comes across IRC. I choose to read when I am available. If I've got a gigantic back load of things that I've not read yet, I just say, "Forget it," and start wherever I am at that given moment.
It is ambient conversation. It's not conversation that should be...if we're talking also too about that decision making, IRC is probably not a great place to do decision making because unless you're paying attention in the moment, it's likely that you won't even notice that a decision was made, or if you're trying to make a decision at IRC and somebody makes a decision, make sure it gets logged somewhere else so that you can actually go back and look at it later.
Slack is better for that because if you've got Enterprise Slack, you can watch and you can read the entire history. You can see some of that stuff. I know there are ways in IRC that you can also do that. Obviously, I've got a proxy set up so I can see the entire backlog. Again, it's like the social existence around those messaging systems where it's not necessary or mandatory for you to read everything. It is really just general chatter.
It's like whatever you would be experiencing while you're in the office, you'll just listen to people talking. I was just sitting over in the breakout room and I could see people just having random conversations, I could hear them doing it. I didn't necessarily need to listen for content. I just knew that they were having a conversation.
It's a way to pass information back and forth pretty quickly but it's also a way to feel maybe a little bit bonded to the people that you're working with even though you're not participating in that given moment.
Gordon:  We've talked about documenting decisions and we need to make certain decisions, make sure everybody in the team knows. What are some of the practices that you use for documenting decisions that everyone has participated in?
Jen:  The easiest way to say that is that if you are working at a product line and you have some sort of roadmap of what you're trying to develop for the year or maybe even a given release. The easiest way to track those decisions is to associate the decisions directly with the work that you're actually doing.
In the space of Scrum and Agile, we call them user stories. We tag them back to problem statements in which we're trying to solve something for our customers. They go through this, not massive process, but a pretty well‑defined process of how we get from problem statement to actual user story where our team is going to do something, but if you are making a decision about, "I'm going to use technology A versus B to solve this problem," and you decide to go with technology B, the easiest place to put that decision is on that user story.
Long‑term, it's harder to capture that information for product line. If you make decision, we're going to go with technology B, how does your customer know? 10 years from now, how do you remember that you actually made that decision? The best way to do that is just to document your product.
I don't really know any better way to do it. Maybe you could have a system to track decisions and you can go back and you can say, "On October 1st made a decision and this is what the outcome was,"
I don't really know how that's valuable if it's not tangentially related to the product that you're working on. Say you're doing a CICD system and you choose Jenkins as your technology, I don't think it matters who made the decision or when you made the decision but rather, was it implemented, is it in the product line, can you document how to use it or what the implementation details are? Those are the things that matter.
Tracking when the decision was made and who made it feels maybe a little bit like you're feeling untrustworthy about who made the decision or why was the decision made. Associating back to the actual work that you're doing is probably the best place for you to be.
Gordon:  Continuing on the theme of tools, one of the things that's been percolating for a long time in business is video conferencing. It would be fair to say that historically a heck of a lot of money has been wasted for not much effect. How do you feel about the current generation of tools and what their value is in terms of teams working together?
Jen:  I don't know anybody who doesn't complain about video conferencing. I was legitimately just at Agile 2016 and listened to some hallway conversations about people complaining about video conferencing. I just took a casual poll where, what are we using? All the big names came up.
Everybody goes through this 5‑ to 10‑minute pre‑conference...there's Internet memes about this. Pre‑conference, who's on the phone? Dialed in, interrupting. All these things have happened.
There are definitely improvement from what I had when I was working and that sole employee, remote, because I had nothing. It's 2016, it should be far better than it is. It just doesn't seem to be any better in the last five years than it was when I first joined Red Hat, really. It just seems to be not going anywhere.
Gordon:  We can all agree that some of the messaging tools, for example, are really quite good now. Without starting a traditional Internet versus software as a service knockdown, you'll find certainly Slack or potentially other tools in that general bane. It has legitimately moved to promote collaboration forward.
I can remember having conversations with Novell maybe 10 years ago when they were doing a lot of work on collaboration. Wouldn't it be nice to be able to do the equivalent of standing up at whiteboard in a room with a bunch of people sitting there and have that kind of interactive, very rich visual experience? The fact is, it's still pretty bad.
Jen:  That's actually a pretty interesting conversation because I've been looking for whiteboarding programs for a while. Just so that we're all clear, especially for folks listening to this podcast who might be creating a for‑pay tool that's really fantastic, I'm looking for free things, things that my Linux users will feel comfortable using.
The space here is not that broad. There are some really great things out there but they're not necessarily perfect. They're still worlds away to go.
I was actually talking to one of my friends a couple of months ago about...I guess they were using these smart whiteboards that they purchased for one of the school systems. I was asking her how it was going. She said it was a complete disaster because no one knows how to use it and so they were immediately broken. They've got these $25000 whiteboards that no one knows how to use.
It was a complete waste of money and time because they don't understand the technology. You can get these really fancy things that unless you really know what you're doing, you don't know what to do. That's sad, right?
Gordon:  Yeah. I'd hate to admit this but probably 20 years ago, having things that take pictures of a whiteboard or that you could use different colors and write and it would supposedly put this onto a computer screen, none of that stuff really worked. I'm sure we spent at least a few tens of thousands of dollars in this stuff at a former employer that never really worked. The really sad thing is, I'm not sure we're an awful lot better today.
Jen:  No. A lot of the video conferencing stuff is always going to be challenged by networking and bandwidth caps. Unless you are somebody who wants to pay a significant amount of money for your home office Internet connection, you're going to be challenged by that. That's not something that I think...technology obviously will make that better as time goes on. Right now, given the state of things in general, it's a goal that's far out in the future to be able to stream video back and forth like that, and not have problems.
Gordon:  I've heard some of the very high end Cisco systems. You have a special room and you have someone directing the cameras. It can be very good. Of course, at that point, pretty much everybody says, "Let's just fly together to a nice place." [laughs] To get there.

Jen:  I have a lot of friends who work at Cisco and they claim that these systems are fantastic. I believe them.