OSpod #28: An Interview with John Dickinson

Jeff Dickey:                Good morning, everyone. I’m Jeff Dickey from Redapt.

Niki Acosta:               I’m Niki Acosta from Cisco and we have an awesome guest with us today. John Dickinson aka @notmyname on Twitter. Please introduce yourself.

John Dickinson:        Thanks for having me. My name is John Dickinson. I work on OpenStack Swift where I am the Technical Project Lead and I’ve been doing that for a while. That’s what I do.

Niki Acosta:               We’re excited to have you. We had Joe Arnold on the show awhile back, definitely somebody technical as well. You’re probably a little bit more involved in the day to day within your role at SwiftStack and your PTL role at OpenStack. I definitely want to know more about the life of a PTL but before we do that tell us about you; how’d you get into tech. Were you the geeky kid with the computer?

John Dickinson:        I was. Yes. That’s how it seems to start a lot of times. Let’s see, how did it all start? It all started when my grandfather gave my family a old Commodore 64 and I started working on that, playing games. The really cool thing about that era of computer was that it came with a big, thick manual. The manual was, here’s how to write a program for this and the computers of that era, you had to choose not to program them.

I naturally got into that and started figuring out how to make this little box do what I wanted it to do, which was cool. The really cool thing is that I’m doing today what I wanted to be when I grew up. I wanted to do programming and stuff like that for decades now. It’s cool because it’s pretty fun and so self taught and school taught and been doing it professionally for about ten years or so now.

Niki Acosta:               How did you get into OpenStack?

John Dickinson:        Almost I guess by accident. At the time I was … Slightly before OpenStack, I was employed but looking for another job and ended up landing at Rackspace. The day that I started at Rackspace was pretty much the day that Cody and Ernest started off to rewrite the storage engine behind the Rackspace cloud files product. We started working on that and worked on it for about nine months and launched into production but a few weeks before we had done that maybe a month or so, the Rackspace executives came to our team and said, “ey guys what would you think about open sourcing this?” I’m like sure, that sounds great. It turns out that that was a part of OpenStack so that became or that was the Swift Project and so I’ve been apart of OpenStack since the very, very beginning.

Niki Acosta:               Will you give us some insight into the history of the Swift name? How did Swift get its name?

John Dickinson:        I don’t know. It was chosen way before I joined Rackspace. It was something that was sketched out in various incarnations. I will say something that this will probably has absolutely nothing to do with how the name actually came up but it’s really a cool fact. Swift is a kind of bird, right? What’s really interesting is Swift birds are the birds that spend the most time in the air at one time. Seriously, the birds spend something like six months flying before they ever land. You can say Swift is the kind of bird with the most up time just like OpenStack Swift has an extraordinarily high amount of up time.

Niki Acosta:               That’s so cool. That’s really geeky and really cool at the same time. On your screen it has your name down there and it has the little Swift Bird and your little icon. I had no idea that Swifts flew that long without landing.

John Dickinson:        It was pretty cool when I figured that out too. I think it’s a complete accident that we’re called the same thing. It’s still a really cool thing and I think it’s something fun that we share.

Niki Acosta:               Go ahead Jeff.

Jeff Dickey:                I want to know more about Swift and the evolution of it to where you started with it to where it is today.

John Dickinson:        It’s really been cool. I’ve been apart of working on Swift for about five years now. The original goal was to rewrite the storage engine behind Rackspace Cloud files to replace something that was already there, to solve some of the problems that it was having with scaling. The marching orders that were given was basically two things, make it better and customers can’t know. In other words you have to support the same API and be able to have some sort of migration plan. We completely solved those things. It was better in every measurable way.

We were able to have a migration plan for all new customers immediately going on there and over time, over the next several months, migrating the existing data from the existing system into Swift. Since then one things that’s been tremendously exciting is that although Swift started as a storage engine in a public cloud hosting provider, something that was very much analogous to S3, it’s continued to grow and meet more use cases for that for different people out there. In addition to being something that is used by many public service providers today all over the world, it is something that is used also internally by a lot of different companies as well.

Those sort of different use cases in those sort of different experiences are things that are really shaped the evolution of Swift itself over the years. We’ve been able to do things that are simple sort of things like, oh great now we can have the ability to say create sign URLs so people can share URLs with content realm. That’s fine. It’s kind of a small feature. It’s really powerful and really interesting supporting arbitrarily a large size objects as the different use cases came up. It also has resulted in some incredibly huge and substantial features that directly is the result of these different people doing stuff.

A couple of years ago we released support for global clusters so that one logical Swift cluster could span continents and oceans. Again, not something that was a part of the original use case but has since been used many times for people who need to have either locality of access for their content or they just need to have their data do-ably spread across a wide area so that they can survive even entire data centers going down. Since then we’ve released something last year called Storage Policies which allows you to directly expose nuances of your particular deployment to whether that’s a locality or region or particular hardware.

You could, for example, say this data is going to be US east coast and US west coast spread and this other data is only going to stay in Asia and the data won’t ever leave Asia. You could also say this kind of data is going to have a different SOA, perhaps a different pricing model because it’s going to be backed by flash storage instead of spinning drives. That’s allowed us to have a lot of flexibility in how deployments look so that a deployment software versus a deployment Rackspace on the public cloud side look very different than say a deployment at Gain Company or CDN provider or some other web or mobile content host-er that’s more on the private storage side.

Recently, the thing is we’ve been to build on top of that again in response to what these people are asking for is support for erasure code content and that’s something we’ve been working on quite a bit. Actually just recently finished up.

Jeff Dickey:                That’s awesome. I know people been asking for that for a long time. Can you describe, too, what that is?

John Dickinson:        Yes. Technology has been around for a long time. I think it was first invented or proposed and described in the 1960s. It’s been used forever. If you’ve ever watched a DVD, you’ve used erasure codes or used a red card you used erasure codes. It’s a way that you can get really high durability, in the case of failures, but not use as much raw storage is required as in a pure replication store to model. In the pure replication model let’s say you’re going to have three copies and you’re going to store those on different hard drives on different servers and things like that, which is really great.

If a whole server goes down, then you still have access to your data. The problem is in some use cases especially when you’ve got large content like backups and stuff like that, if you’re going to store one gigabyte of data you’ve got to store it three times so that’s three gigabytes that you have to store. With erasure codes you can break it up into some smaller pieces and compute some other what’s called parody pieces and store those. Maybe you’re effectively only storing, say, 1.6 or 1.8 times the amount of storage. You only need 1.8 gigabytes to store your 1 gigabyte piece of data.

You can still be protected against many failures when you’re doing that as many or even more failures than in replicated storage. The cost, though, is that it takes a lot of CPU horsepower to do this. It’s not particularly good for every use case but it is really good for those use cases where you have this large set of data that needs to be stored fairly cheaply and you’re not going to be accessing it frequently. We’ve been working on that inside of Swift for the past year or two depending on how you’re counting and literally in the last 48 minutes we have been living in a post EC world in Swift. Just late last night and early this morning to get all of that stuff merged has been a truly global effort from a lot of really talented and great to work with contributors in Swift community.

Niki Acosta:               You’re a PTL and you have been for quite some time I think, right?

John Dickinson:        Yes.

Niki Acosta:               What is that like? I think a lot of people just think this is the guy that’s going to say no to my code or … You actually do have the ability to make decisions but you’re also held accountable by the community, right?

John Dickinson:        Sure. Yes. It’s an interesting balancing act because inside of an open source community there’s not … I would say it’s very similar and OpenStack itself, to be honest, is very similar to the organization of a large company. You’ve got different levels of bureaucracy and you’ve got different levels of people talking to each other in politics and all that kind of stuff. The difference is that you don’t have nearly as much accountability within that as you would, not to put too fine a point on it.

You can’t actually hire and fire people inside of an open source community. Which means that getting people to work together is much more along the lines of making sure that people have the tools they need both to get their work done but also to know what is being done by other people and how to take that to their employer and tell that story to their employer and to show this is why the community is good and this is why we’re working on these sort of things because it helps us over here. It means that there’s a lot of time spent on making sure that people aren’t stepping on each other’s toes.

They’re working together and not against each other and that the things that are being focused on are actually things that are moving the entire project forward rather than people going in a hundred different directions at once. At least in my experience the life of a PTL involves a lot of that sort of thing. It does still involve a little bit of writing code, not as much as it used to. It does still involve reviewing code and making sure that I can be available for anything that’s necessary there.

It also involves a lot of talking to people and making sure people know what’s going on and even outside of the developer community even just the immediate Swift community but the broader OpenStack community and even the ecosystem as a whole figuring out that okay we know that, for example, HP is seeing this issue and has this concern and needs to work on this. Then we see that Intel is over here doing this and Red Hat’s doing this and SwiftStack is doing this and so on and so forth.

You’ve got all of these different people together and it’s been really fun building the tools and the processes and staying involved to help people out. Everything from tech support to evangelism to actual typing code sometimes and making sure the other people who are primarily doing the code writing, reviewing are able to do their job with as few distractions as possible.

Niki Acosta:               We didn’t define PTL for people, that’s my mistake, first. For someone who’s not familiar with that term, can you explain what that is?

John Dickinson:        It stands for Project Technical Lead and the basic idea is that it is inside of all of the OpenStack projects in some ways it’s the point person, it’s the face of the project to the rest of the community. It’s someone who’s elected every six months by the people who are contributing to that project. It’s chosen from a group of your peers and by the community in order to … In some cases set directions and priorities but in a lot of cases to coordinate and to make sure things still happen and to be that face and voice in the community. The person who gets the short straw, every time somebody needs to stand upon stage and say something or type up things or be kind of the person who if there needs to be a tie breaking decision, the person who could do that and the other people will trust and respect to do so.

Niki Acosta:               That’s not a long tenure for people. Six months is not a very long time. How do you grapple with staying in your position of PTL? Do you have to be the ultimate nice guy and make a lot of friends or is it based on technical merit?

John Dickinson:        I think it’s a lot of both. In some cases I’m fortunate to … Nobody else has wanted to do it so I’m the one who’s, okay, I don’t mind the tedious parts enough as much as I enjoy some of the other parts so it’s … To be honest I haven’t there’s been nobody else who’s stepped up to say yes, I’d really like to do this. There are many people in the community who are very capable and able to do it if they chose to do so but for the time being they are happier not focusing on a lot of the external facing but focusing on the inward what’s happening to the Swift code itself. I got to run for my job every six months but so far it’s been a joy to do and not a burden and something that I’m proud to do and really honored to work with all of the people in the community. If there’s anything good that’s happening in Swift it’s not because I’m PTL, it’s because of the awesome contributors.

Niki Acosta:               That’s how you keep your job, ladies and gentlemen. You’re such a good representative for your project. How much of your time … I certainly understand that Rackspace was super open source friendly. Obviously as a founder of OpenStack they were, yes, go be a PTL and at SwiftStack you have that same luxury. How much of your time do you spend working on stuff for SwiftStack directly versus PTL duties?

John Dickinson:        I spend 24 hours on both! The really great thing about being at SwiftStack is that my duties as PTL do not conflict with my duties as a SwiftStack employee. We are very focused on providing an awesome storage product for our customers and to make sure that people’s storage problems are solved. One of the key pieces of doing that is to stay involved in the open source community so that we know that our customers have a voice inside of the project itself and that we are also able to bring to our customers a high quality software developed by world class engineers all over the world. The two work together very, very well in that and I’ve yet to find any conflict of interest or any sort of tension between the two. Occasionally, it gets busy but for the most part it’s a joy to do both and by doing – working for one – I’m also working for the other at the same time and there’s been no struggle there. It’s been great.

Niki Acosta:               A lot of what you guys do, if I remember correctly based on our conversation with Joe Arnold, pretty much all of the stuff that you guys do is contributed back, right?

John Dickinson:        Yes.

Niki Acosta:               Where do you draw the line between your IP and the stuff …

John Dickinson:        The truth is we sell software so if we’re giving it all away how are we making money sort of thing. Anything that affects the actual storage engine and pretty much what you’re talking about is the read and write data path for the object storage system, we contribute that. That’s everything from performance improvements to security fixes to new features – things like global clusters and ratio codes and things like that that we’ve been able to take a leading role on.

Beyond that though I think it’s important to consider that Swift, like all of the OpenStack projects, it’s not a product; it is a project. It is an open source code project and it’s almost unfair to compare OpenStack projects to other products that are out there in the world, in the wild, because a product has a lot more around it. A product has a lot more polish around it. It has a lot more or the operational things worked out. It has a lot more of the deployment things worked out and it’s needed … you’ve got a lot of those integration points that you got to deal with in a product.

Those are the kind of things that we do for Swift itself to bundle up a product called SwiftStack. To do that, we have spent a lot of time on working on the integration with the rest of the IT infrastructure so making sure that it works with your existing identify management system so you don’t have to have something brand new whether that’s an old app or what not. Being able to integrate with charge backs and billing systems so that a company can easily see who’s using what and monitor that and appropriately measure that.

Just figuring out what’s going on in a cluster with alerting and monitoring and metrics and displaying those so that operators know what’s the state of your cluster and what should I do when something happens. Know matter who you are and know matter what’s going on, if you’re deploying any storage system you have to have those sort of tools. If you’re doing Swift, you’re going to have to have that information. You need to know what’s going on. While Swift has a lot of data that it tells about itself that you can hook into to figure out what’s going on, putting it all together and integrating it into the rest of the systems that you may have is exactly what SwiftStack is doing so that you can actually focus on building apps instead of worrying about your storage system. We spend a lot of time on that sort of thing.

The question I guess about where do we draw the line, if it’s about the object storage system, completely hundred percent open source. There’s no lock in on that and we are using upstream Swift code. We don’t have our own special version of Swift that we’re doing. If it’s talking about the operation tool sets and the monitoring and say like a file system gateway on top of that, those are the type of things that we have as part of our product that our customers have access to.

Niki Acosta:               In terms of Swift, a lot of the criticism that I’ve heard is that it’s really freaking hard to … it’s not that hard to install but it’s really hard to scale. Do you think that’s true? I mean you’re a Swift expert but do you think that’s slow deduction?

John Dickinson:        I think that’s a good question. That’s an interesting thought. There’s two ways you can take that. One is the question on does Swift itself scale and the answer is unequivocally, yes. That’s not just me saying that as an advocate for Swift. That is me saying that with the backing of many, many, many multi petabyte deployments on Swift some of whom are very large including people like Rackspace and HP and even other private ones. The truth is, yes, Swift is a very massively scalable object storage system.

Any kind of struggles you have with scaling Swift and managing that, in fact, at times I’ve heard somewhat of the opposite is what the great thing is is once you got it installed, it’s very easy to add new capacity and you don’t have to have any sort of down time to do that. Yes, you are dealing with software that’s running across a cluster of machines so yes there’s operational and management needs that you have there. That’s again unplug is part of what we do at SwiftStack.

That’s absolutely something that were very concerned with in the open source project as well to make sure that the operator’s needs are accounted for and that we can always maintain things like great we’ve got a stable API and you’re always going to be able to upgrade with no downtime. You’re going to have a stable release that is going to not frequently update API versions or config variables and you’re always going to have a migration plan from one to the next.

Niki Acosta:               With the customers that you’re working with, are you seeing that people are migrating data to Swift from something else?

John Dickinson:        Yes.

Niki Acosta:               If so, what are they migrating from? What is Swift meant to replace or augment?

John Dickinson:        That’s a really good question. The answer is yes. There is … A lot of people are migrating to Swift away from something else. I’ll talk about what those are in just a second. In general, let’s talk about what is Swift good for? Why would you … What are the use cases that you would actually use for it? There’s lots of other projects that people ask about and say well how does it compare to this other storage system X, Y or Z or whatever it may be. Swift is designed for storing unstructured data. What’s that?

Think about … The common thing that we think about a lot is files. Think about documents. Think about your pictures and videos. Think about stuff you see on the internet. That is unstructured data. Basically it means that if you and I, Niki, are using the same, for example, phone app and we each take a picture. It doesn’t really matter. Those are completely unrelated pictures. It doesn’t matter where we were when we took those or what order those specifically happened in, they are unstructured with relative to one another. In aggregate all of our pictures together. It’s just a big bunch of unstructured data. Another really common use case and one of the things people start using Swift for initially is backups. I’ve got a server, I’ve got a lot of servers and many times it’s using something like Nova is I’ve got all of these servers and need to back them up so where do I put all the backups for that server? You can take a backup of a server, you can put that inside of Swift and now you know it’s going to be available when you need it. Hopefully you won’t. If you do, it’s right there and then it’s also going to be durably stored so that you’re not even going to lose it if you have hard drives go down or servers go down or need to replace that over time.

When you’re asking the question of where are people migrating from, many times it’s people who have been trying to solve these problems already and have run into scaling problems either technically or financially with traditional storage systems. There is definitely some degree of migration away from some public cloud providers like Amazon S3 or something like that. Many times people will start balking at a bill that is measured in tens of hundreds of thousands of dollars a month. It’s dominated by a network costs and maybe they don’t even want to give another company their data so they need to keep that in house because that’s their … that’s actually key to their business.

In that case there’s people who are using Swift for cost savings over public cloud storage. On the other hand, there’s a lot of people that I see, especially people who are doing more of the private cloud thing, who are migrating away from traditional storage providers, NAS and SANS, things like that that really have inelegant scaling and upgrading models and get very expensive overtime – especially when you’re locked into some sort of support contract or something like that. I see all of that and that’s much more common than the general, I think I’m going to have this idea and it’s going to store a lot of data. I guess I better deploy Swift and do this green field deployment.

To give you a really practical example or a few really practical examples. One use case of Swift I like as far as public web content that we talk about a lot is Wikipedia. If you go to Wikipedia and you look at a picture it’s coming from their open source cluster that they are running. You’ve got this web content that is incredibly popular one of the most popular websites on the entire internet and the data has to be there. By the same token looking at those pictures and videos, you’ve got businesses whose livelihood is – requires that that’ll be there like online auction sites.

If there’s not a picture of whatever you’re trying to buy you’re not going to buy it. They’re using Swift to reliably store that data. There’s a lot of people who are using Swift for media and entertainment, videos and pictures and games recently talked about CDN provider who’s using Swift as an origin for a lot of the content that they’re doing especially that’s popular in the gaming industry, financial industry is really sensitive about where their data lives. A lot of them are saying I’ve got all of these documents and I can’t give them to a different company so I have to make sure that they are safely and secured in my own data center and I need a way to do that scalably. They’re using Swift for that.

Niki Acosta:               Does SwiftStack have any opinions on the actual underlined hardware that customers can use?

John Dickinson:        Yes. Swift itself is hardware agnostic but we … That’s one of the things that we do here at SwiftStack is definitely work with other companies like Redapt to – Hi Jeff – to make sure that our customers have a set of hardware that is tailored to their use case and actually is something that we can support well and is just to make sure that they’re successful.

Niki Acosta:               What are your baseline hardware recommendations? Are you putting a ridiculous amount of drives in commodity hardware or does the [inaudible 00:30:44] matter?

John Dickinson:        In a lot of cases, yes, that’s what it looks like but I think the important thing there is it’s generic off the shelf hardware. It’s not any sort of customer hardware. If you’re preferred vendor is HP, good you can use HP. If it’s Dell, you can use Dell. If it’s Cisco, you can use Cisco. You can do all of that kind of stuff and you can tailor it for your exact needs. In a lot of cases, yes. It does look like I’ve got a box with a little bit of CPU and a whole lot of drives. That’s actually one of the fun things that I get to do is I get to talk to these guys who are these storage media vendors.

I’ve got a box of 8 terabyte drives sitting on my desk that I’m about to rack up for a testing cluster and I’m going to get another box of those in the mail pretty soon from a different vendor. The really great thing is that as this new storage media is coming into play, we get to make sure that Swift is keeping up with that and actually we’re being approached in the community by these people who are making the underlined storage media to say hey, we want Swift to work natively and well with this out of the box, the moment we launch stuff.

Not only new technologies with hard drives but flash vendors and even recently there’s been some noise about a couple of companies that have written tape library connectors for Swift so that if you need the ultra cold ultra large scale storage archive data sort of thing, there’s people out there who are actively working on that right now in the ecosystem. It’s been really exciting. That’s actually what’s really exciting me a lot right now in the open ecosystem is being able to work with the storage media so that Swift can natively talk to those very, very well and ultimately whole thing is about applications and application developers really don’t care what kind of hard drive it is, they just want to make sure the data is stored.

Being that abstraction layer between here’s what the client is talking about, the client doesn’t have to think about the hard problems of storage anymore. The being able to then expose as needed the nuances and the take advantage of how the storage media vendors are pushing the envelope technologically, it’s just a really cool place to be to see the intersection of the latest phone app, the latest gene sequencers talking into Swift and dumping the data into their storage where on the backend side of things, we’re taking advantage of 8 terabyte and larger hard drives and tape libraries and dealing with flash storage and it’s just really cool. It’s really fun.

Niki Acosta:               Is that done in the driver model like it is with Signatron or some other projects?

John Dickinson:        In a sense, yes. There is a plug them up model inside of Swift that you can do that. I would say it’s subtly different from the way Nutron does things and some of the other open stack projects in that one of the things we’ve been very focused on inside of Swift is that we are actually the complete storage engine implementation. Swift is not passing the data to some other durable storage system but we are actually looking at the media themselves. One of the ways that we’ve seen or a few of the ways we’ve seen this used in the ecosystem already is, for example, being able to talk natively to see its new connectic platform which is a whole new way to talk to hard drives. It’s a key value store rather than using say the commands.

Like I said, we have the tape library people who are using the same abstraction models to talk to tape we’re talking about optimizing for flash storage using the same things. I know that for example HP has developed some stuff to work with some of their proprietary storage hardware and all of that stuff together is, yes, the ecosystem is vibrant and there is a lot of ability to write a plugin that works with your particular system.

Niki Acosta:               Are you guys just running Swift as a standalone or are you also deploying maybe some other components of OpenStack or other projects of OpenStack?

John Dickinson:        When you say you I guess you’re referring to SwiftStack there.

Niki Acosta:               Yes.

John Dickinson:        We are not incredibly opinionated about that really. We are wanting to make sure that your storage problems are solved and as such, if you want to deploy Swift completely standalone and independent from anything else including Keystone and Nova and Cinder and all of that stuff, that’s great because we want to make sure your storage problem is solved. However, if you already have an deployment that includes other OpenStack projects then, yes, we’ll happily make sure that we work with those and that you can talk to Keystone and that we have the right pipeline set up so that it is well integrated. Yes, Swift in general is very well integrated with the other OpenStack projects; but, as well, can be deployed on its own. My opinion there is if you got storage problems, let’s solve those and I’m going to let somebody else solve the compute and networking problems that you might have. I’m going to focus on this to make sure it’s done well.

Niki Acosta:               What is the breaking point when someone would say I would actually save money looking at SwiftStack instead of Amazon S3. Where does that price break happen understanding there’s probably other benefits that security, data, location all of that stuff, but if you’re spending X at Amazon you should probably look at Swift like what is that X?

John Dickinson:        I think that X is wherever it hurts. When does it hurt for you because we can make that number smaller almost most likely, almost definitely. We’ve seen everything. We’ve talked to people who have Amazon bills that are literally seven digits a month and we have customers who are nearly tens of thousands of dollars a month. With that being said, you could go to Swift.com right now and sign up for a free trail right now and do that on one machine with just a few hard drives and take it for a spin and see what you think about it. In general though, most people are going to start in a production setting with a few boxes. If you think about a few boxes, each having probably twelve or more drives, then you’re generally looking at a few dozen terabytes as far as a getting started place and then going from there. Then, of course, when people are getting to multiple petabyte deployments, that’s when it really starts getting attractive.

Niki Acosta:               At what point do you say okay we need to spend up another region or availability zone or whatever you call it? What is the max deployment site before you go okay it’d be in your best interest to move that over? Does it depend on use case?

John Dickinson:        Of course everything depends on use case. It all depends. Since Swift has a very highly modular architecture in that there’s the data access layer and then there’s the actual storage layer and you can optimize those independently, it means that if you need more you can add more wherever that is. If you need more client connectivity and that’s your bottleneck, I just need more gigabits per second to serve content, then you can expand that part without also having to invest in a bunch of hard drives that you don’t really need today.

On the other hand, if you’re doing okay on the client through foot part, but you are running out of storage space, then you can buy some more hard drives and plug those in. You can basically tune the cluster exactly to your use case and I think that’s one of the really powerful things about how it works. If your use case is that you need to have two data centers, great you can do that, whenever you need to whether that’s immediately at the beginning or you want to move into that over time over the next whatever time frame you have. If you don’t need that, you don’t have to do that. It’s a very highly customize-able system so that it can exactly meet the use case that you have.

Niki Acosta:               I know in Metacloud– now Cisco OpenStack Private Cloud– one of the decisions that our founders made, which I think wasn’t a bad decision, it was to make sure that everybody was on the exact same code base. Do you guys follow that rough same model and if so how are you guys handling up rates for your customers?

John Dickinson:        I’m glad you asked. We’re handling it very well, Niki, thank you.

Niki Acosta:               You didn’t ask me to ask that question, by the way. Just to let you know.

John Dickinson:        I know. When you’ve got a company especially something that is not a giant behemoth company that has thousands and thousands of people in their support departments alone, then, yes, making sure that your customers have a reasonable and supportable platform is really important to both your success as a company but also to your customer success so that you can actually make sure that they have something that’s going to meet their needs. There’s a couple of things on that. One, is that we follow and make available to our customers the current development inside of Swift.

Swift generally ends up releasing a production ready to release about every six to ten weeks depending on what’s going on in the community. We make those available to our customers very, very soon. As I said, one of the things we keep very near and dear to our heart and we at SwiftStack are along with the rest of the community pushing for in the community is to make sure that we have good migration facts. When there’s no available version of Swift inside of the SwiftStack products, yes, you click a button and we go through and do a rolling upgrade of the cluster with no client downtime in a way that’s validated so that you know that you’re not going to be introducing instability into the cluster.

We’re taking a no down. We upgrade it, we bring it back up, we test it to make sure it’s good then we can add it back to the cluster. We do that on the next note and we do that on the next note. We go through the entire cluster in a very seamless upgrade so you just literally click a button and you’re done. It’s off loaded those hard operational tasks so that you don’t have to worry about those.

Niki Acosta:               Are you guys using some kind of custom automation tooling or are you using something that’s off the shelf?

John Dickinson:        I would say that we’re using a lot of other open source projects that are out there to coordinate and to establish that but I would say it’s a lot more complicated then oh we’re just using puppet or chef or ansable or something like that. The reality is though that we’re using some of those kind of components, of course, but to be able to automate that and put it together we have developed our own tooling so that we can effectively and securely do that reliably.

Niki Acosta:               So cool. That’s a feature I think definitely talking to you customers in all sizes and you talk to start ups and it’s like we got this figured out because everything is greenfield, everything’s great then you talk to Enterprise and they’re like what is this tool? Is it validated? Does it check the box on the RFP? I’m starting to see the tides change for sure. People are definitely like okay yes we decided we want OpenStack now we have to figure out which way we’re ready to take it out of the lab and do something serious with it. I do see the tides sort of shifting there. You guys seeing the same in terms of the adoption?

John Dickinson:        We’re very much focused on more of the storage side of things than say oh we’re going to follow the latest OpenStack hype or market releases or press releases or things like that. While we are very proud members of the OpenStack Foundation, what we’re following there is where are our customers who need storage and a lot of times those are coming from the more traditional storage vendors. Yes, we’re seeing a huge change in the industry overall in that people are looking for object storage solutions so that they can have scalable distributed and durable storage. It’s really driven by a change in use case. I mean the fact that we’ve all got these super computers in our pockets that are taking pictures and uploading content but also when you want to make it available to your employees. Can I tell you a quick, fun story on that?

Niki Acosta:               Yes. Please do.

John Dickinson:        A really cool use case. There’s a trucking company on the east coast and they were giving a talk at one of the OpenStack Summits a while back. It was called Bud Van Lines and something I’d completely never heard of but they are a moving company like big 18-wheelers going down the interstate and they go to your house and they pack it all up put it in there and they take it to your new house. That kind of thing. Ultimately, they are one of these traditional, boring companies. They’re not the next new startups or things we tend to talk about here in Silicon Valley. They have huge logistical problems. That’s actually what they’re solving.

You get somebody’s who’s pulls up in a truck and they’ve got to move your house, your stuff. In a lot of ways this is the kind of stuff that is defining who you are. This is your pictures, your family heirlooms and mementos, your furniture, where you sleep every night. It’s really important that this stuff is safe and secure and not broken and all of that kind of stuff. One of the things they can do is they can take a tablet into the house and just go oh there’s something there let’s take a picture of it. What it look like before we moved it so when they unpack it they can see what’s going on. Guess where all those pictures are stored?

Niki Acosta:               Swift.

John Dickinson:        SwiftStack specifically. Then they put everything into the truck. They’re not necessarily going to be on the same day at your new place so they’ve got to go park it a a warehouse overnight and you want to make sure no one’s literally stealing your house while it’s in a truck someplace. You’ve got video surveillance on that 24/7 to know exactly who’s coming and going, watching the trucks, watching the loading docks, all of that kind of stuff. Guess where all that video is stored?

Niki Acosta:               SwiftStack.

John Dickinson:        It’s even better than that because that’s pretty cool in and of itself. What’s really cool about it is that then you can anybody else there can say what’s the status of this? Let me pull up the order. Let me see what’s going on. There’s a picture of that dear Aunt Gertrude’s heirloom vase and, yes, there’s a picture of it and here’s what’s going on. I was talking to one of the employees there last November in Paris at the OpenStack Summit. He said you know what, yes if I wanted to right now order a restaurant in Paris I could see what’s happening and pull this up and do that and look at the video, look at the pictures. That’s the kind of use case that’s … those are the kind of changing application use cases that I think are driving this change in we need something new for storage and we need to be able to actually have it available right now. It’s growing at this massive rate so we got to have something to scales. We need to be protected from hurricanes that hit the east coast and the gulf coast and earthquakes that hit the west coast. We need to have geographically distributed that. That’s the kind of thing that Swift is built to solve. That’s the kind of thing that we’re solving for our customers.

Yes, I guess the long way around here, absolutely, we’re seeing that the ecosystem change because having a few dozen hard drives on a proprietary storage box sitting in your data center is not sufficient. It’s not cost effective and what you need is something that is a modern object storage system to solve those needs.

Niki Acosta:               I just had a flashback to the John Oliver episode on Surveillance where he interviewed Edward Snowden.

John Dickinson:        Yes.

Niki Acosta:               I can’t imagine how big the NSAs storage cluster is given the fact that some of their people have spoken at OpenStack Summit in the past, I’m pretty sure they’re using Swift.

John Dickinson:        I have no official knowledge of anything like that.

Niki Acosta:               Great, now we’re both on a watch list. All three of us.

John Dickinson:        No kidding.

Niki Acosta:               That’s all you have to say about that.

John Dickinson:        I would say that look, Swift is a really great tool to store large amounts of data. I do not condone every potential use case at all. There are a lot of use cases that are out there and I think they are a lot that are actually have the opportunity to do good in the world whether that’s storing data that is taken from satellites to making sure people have, can share photos to watching videos to just having reliable backups. That kind of stuff is really important and that’s the kind of things we’re looking at.

Niki Acosta:               It’s really cool. It’s so refreshing to hear excitement about Swift. Personally I don’t think it’s boring. It’s been around forever since the beginning and there’s all these new projects and Swift is still there chugging along. It’s not as exciting because it’s not one of these shiny new things, but man it works.

John Dickinson:        It works. It …

Niki Acosta:               Nova works. Nova is pretty solid. Neutron & Ceilometer, still variable. Swift and Nova, they’re solid and I don’t know if it’s just the maturity point that they’ve reached or they were just in a good place when they were open source to begin with .

John Dickinson:        I think that the thing that I would point to it’s easy first thing to point to is that Swift is good because it’s tested and run in production, which, yes, of course, that almost goes without saying. The true reason and the reason that Swift is really great is because of the contributors behind it. The people who are getting the code, the core viewers inside of Swift are some of the smartest and most mature and intelligent and pleasant people I have ever worked with. It’s been absolute pleasure. When we’ve been finishing up this literally like I said just an hour ago we’ve finished merging the emerging code data support into Swift.

That is the culmination of two years worth of effort that was done from Paul and [inaudible 00:50:58] at Intel across both the United States and China. We’ve got Tristan and Chiago from Red Hat in Germany and the United States. We’ve got Matt in Australia who is working on that. We had Koda and Itashe in Japan who were reviewing a lot of that work. We’ve got Sam and Clay and I here at SwiftStack in California and it’s just been this truly amazing … HP we got Allister who’s over in Galway – no he’s not in Galway, he’s in Bristle in England but working with their team in Ireland.

That truly global reach and the quality of those people are really what is making Swift an incredible stable and reliable and mature piece of software. It’s a pleasure to work with all of those people and that is why if there’s anything good that people say about Swift it’s because of these people behind it and it’s been really great.

Niki Acosta:               They say that diversity results in motivation and I can’t think of a better example than probably what you guys are doing at Swift or just OpenStack overall. When you’re a customer selling a product to a subset or a group of customers that look alike you can’t potentially account for every use case under the sun but that’s something that OpenStack has to solve for sure. It’s run by vendors and blah, blah, blah and the startups aren’t getting funding and all those people that you mentioned are employed by somebody and those companies are basically paying their employees to contribute to OpenSource which is also very cool.

John Dickinson:        I think something that’s pretty interesting that we’ll see in the community more and more and you’ve already seen some of it happen in the OpenStack community right now is the technical problems are secondary. Some of those are pretty tricky but the community problems are the hard part. People are both the problem and the solution. Getting people to work together and insuring that we have viewpoints from all places, all walks of life and we know that that’s what gives us a healthy community that we can stand behind and actually allows us to produce things that are actually useful for people because that’s what it’s all about.

It’s not just I got this interesting idea and something I can put on my resume. No, there’s people that we need to solve problems for and that’s what we cannot forget and the only way we can figure out how to solve those problems is we actually listen. Getting involved in there is us and the community. Not necessarily just the people who are going to contribute code to us but also the people we are serving by writing code. That’s what’s truly important and we have to keep that.

Niki Acosta:               That’s so inspiring and at the same time I can’t help but wonder if your proximity to Haight-Ashbury is starting to rub off on you.

John Dickinson:        That’s on the other side of town.

Niki Acosta:               You’re a little different than you were back in the Rackspace days. That’s what experience does for you, right? I remember how sad we were that you were leaving. We were like oh John’s leaving Rackspace but we were still happy for you.

John Dickinson:        It was a fun time being there and definitely great working at Rackspace.

Niki Acosta:               How is that transition from Texas from San Antonio to San Francisco been?

John Dickinson:        I love it. I love living out here in San Francisco. I love the weather. I love the culture. I love the geography and it doesn’t hurt that I’m in technology and it happens to be the current capital of the technology world. It’s nice but it’s pleasant weather and I can bike to work and I have great views of downtown at my office window. It’s fantastic.

Niki Acosta:               That view does not suck. We saw that when Joe was with us. You guys had just moved in I think and have like a …

John Dickinson:        We’ve only been here a few months so it’s pretty new but it’s great.

Niki Acosta:               That’s so killer. We’re running close to time here. My co-host has taken a back seat today because I’ve been getting super excited about Swift. Jeff, do you want to do the honors?

Jeff Dickey:                I do. Can I ask one more question? I know we’re running over.

Niki Acosta:               Yes.

Jeff Dickey:                I just want your thoughts on block storage and what do you see next and how is block going to work?

John Dickinson:        That’s a good question. You should ask somebody who knows something about block. No, I think that block storage is pretty interesting and it’s a crucial use case for overall storage. With the stuff that I work on, I am completely not trying to solve that use case. It’s got a radically different set of requirements and usage models and things like that. You’re generally looking at stuff that is really close to the hardware, super performance because you need to be able to get the last ounce of performance from that. You’re generally looking at things that aren’t as quite as large of scale.

You’re not measuring block storage systems into petabytes and exabytes and beyond. You’re measuring it generally into terabytes scrapping into petabytes. I think it’s really great. I’m really excited to see what’s happening in the flash world and the new innovations there. I don’t think there’s any doubt that that is the future and what is happening especially that kind of storage and really just almost all kinds of storage. I think it’s great. I’ll be happy to consume that to just as a new user but I don’t have a lot of technical knowledge or it’s certainly not what I’m trying to solve today.

Jeff Dickey:                Cool. We thank you so much for coming on with the last question we like to ask is who are two people you’d like to hear on the show?

John Dickinson:        I know you told me when we were talking about prepping for the podcast, we’re going to ask you this you got to come up with some names and you know what I didn’t come up with any names. What I did instead is I think I looked back over the kind of people you’ve had on the podcast and it’s been really great especially a lot of the people in the OpenStack Ecosystem who are involved with the – especially on the business side of things and driving some of this strategic direction around a lot of the member companies and I think that’s really great.

What I would suggest is having on this podcast would be some more of the OpenStack PTLs and hearing from the horse’s mouth, sort of speak, what is actually going on in the projects and what’s coming next and how things are working more on that half of the house. That’s what I’d say is get some of the OpenStack PTLs, some more of them on here.

Niki Acosta:               I would be like make sure you watch John’s first because …

John Dickinson:        Now you know what to do next time.

Niki Acosta:               … he’s the people in the project. He’s slick. How can we find you? You blog on the SwiftStack website.

John Dickinson:        I do. Generally the best way to find me is on Twitter @notmyname also I got a little website calling card @not.nm and you can find my blog in Code Repos and stuff there. I’m pretty active on SwiftStack’s blog. You can find us at SwiftStack.com. Then on IRC. I’m always on RFC. My nickname on freenode is notmyname.

Niki Acosta:               Thank you so much, John, for taking the time to talk to us today. I know you’re juggling being a SwiftStacker and being a Swift PTL. We are definitely looking forward to seeing you in Vancouver. Hopefully before then I’ll respond to your email and you’ll help me make my first co-contribution.

John Dickinson:        Yes, I’m working on that. I’ve been focusing on the EC stuff but that is getting towards the top of my list. I want Niki Acosta to be a Swift contributor.

Niki Acosta:               Yes. I can’t wait. Jeff, who do we have next week?

Jeff Dickey:                Next week we have Jay Pipes from Mirantis.

Niki Acosta:               Jay’s at Mirantis? Holy smokes I need to check my email, huh? Cool. We’re looking forward to talking to Jay next week. Thank you everyone for joining us. Tweet us. We know there’s a problem, by the way, with iTunes. We’re getting it fixed so thank you for your patience and thanks for bringing that to our attention. Let us know what you like, what you don’t like and until then, we’ll see you next week.

Jeff Dickey:                Thanks everyone.

Niki Acosta:               Bye.

Jeff Dickey:                Bye.