Understanding MongoDB transcript

now we're going to talk about mongod DB mongodb is pretty popular in corporate environments and uh you might find the need to integrate it with your Hadoop cluster or with spark or something like that it's easy to do so let's go find out what mongod DB is all about what's different about it and how to integrate it with your hop cluster let's talk about mongod DB next mongod DB is a popular choice in the corporate world in particular because it is built by an actual corporation that actually supports it as opposed to just being kind of out in the wild and open source mongodb's name comes from humongous data humongous get it which is kind of weird because really what sets mongod DB apart is not the fact that it can handle big data but just its document uh data model it's document based data model which is very flexible we'll talk about that more shortly so don't let the name fool you other nosql databases do just as good of a job at managing Big Data where are we in the uh triangle of the cap theorem here well mongodb sits down on the consistency and partition tolerance side of that triangle so since it does have to deal with big data partition tolerance is something it just has to do and mongodb chooses to favor consistency over availability so mongodb has a single Master a single primary database that you have to talk to all the time to ensure consistency but if that Master goes down it will res result in a period of unavailability while a new primary database is put into place so the big thing that's different about mongod is that you can stick pretty much anything you want into mongod DB basically any Json blob of data you can shove into a document in mongodb it doesn't have to be structured you don't have to have the same schema across each document you can put whatever you want in there so here's an example of what a actual mongodb document might look like let's say that we want to store blog posts in a mongodb database well this is what it might look like and this is really what it looks like so mongodb will automatically give you an underscore ID fi that's just automatically append to your document that contains some unique identifier for you and that's done because there is nothing in mongodb that says that you have to have some unique field in your document at all so within that document we might have a title the content of the blog post itself and then we can have a comments field that contains an array of other documents so this is an an example of an embedded document where we have a document representing a com comment that itself contains a name email content and rating and I could actually have multiple of these embedded within this blog post document so that's a little concrete example of what a document might look like in mongod DB like I said no real schema is enforced in mongodb at all you can have different fields in every document if you want to uh don't obviously not necessarily a good idea if you want to actually do fast lookups in that database but you can you're not you don't have to have a single key value like you would have to have in Cassandra that's some unique identifier but you can create indices in any fields that you want you can Al also create indices on combinations of fields so one nice thing about mongod DB is that it's very flexible in how you can index its data to uh achieve fast lookups on whatever queries you might be doing obviously if you want to actually Shard your mongodb database which is how they talk about actually horizontally partitioning it so that you have different ranges of data on different servers then you have to have some unique index to do that charting on and we'll talk about that more in a bit so with mongodb you have a lot of flexibility and what you can store in it but with great power comes great responsibility just because you can shove whatever you want into mongodb doesn't mean you should you still need to think about what the queries are you're going to be performing on this database and design your databased schema accordingly so make sure that if there are think about what indices you might need for fast lookups for the queries you're going to do at the end of the day it's still a nosql database so you cannot do joins efficiently so you want to make sure your schema is denormalized as much as you can in mongodb world we talk about databases and Collections and documents instead of databases and tables and rows so this kind of gets away from the notion of there being some sort of fixed schema which is kind of implied in the words table and row so a mongodb database contains Collections and a collection contains a collection of documents so instead of tables containing rows we have collections that contain documents conceptually you can think of them the same way but just keep in mind that collections can contain pretty much anything and the main restriction here is simply that you cannot move data between collections across different databases so if you do need to reference data between different collections they do need to be within the same database if I can editorialize a little bit here if you uh go to the mongodb website you'll see it's really aimed at more of a corporate environment and uh I don't know kind of rubs me the wrong way to be honest if you look at the about mongod DB tab for example it doesn't really tell you anything concrete it says with mongodb these organizations move faster than they could with relational databases at on10th of the cost with mongod DB you can do things you could never do before wow that sounds great to you know the sort of CTO that hasn't written code in 20 years right but for technical people like you and me not really very helpful kind of Rubes me the wrong way but for corporations this can be a good thing you know you want to be able to pay for a professional support and have guarantees about support if you need it so you know mongodb has that sort of service available to it and at the end of the day it is still open source and you can still get the documentation you need as a developer if you just go looking for it but um man websites like this just really bother me all right let's talk about mongod DB's architecture so the first thing you need to understand with mongod DB is what they call replica sets so like we said before mongod DB has a single Master architecture the idea being that we want to have consistency over availability but you can have these secondary databases that maintain copies over time from your primary database so as Rights happen to your primary database those rights get replicated through an operation log to any secondary nodes that you might have attached to it so in this Doc in this diagram here we might have a primary mongodb server that your application talks to and maybe we have a couple of secondary backup nodes in one Data Center and a couple of secondary back up nodes in some other data center mongodb will automatically replicate those operations to those second AR so that in the event that the primary goes down one of these secondaries can take its place and the the way that replication chain works is kind of arbitrary uh it actually just tries to figure out which server can it talk to most quickly you know where's it getting the fastest ping times from so you know you don't necessarily have this sort of structure where you have a primary talking to a secondary and another secondary backing up from another secondary these arrows could be pointing pretty much anywhere in practice so the good thing though is that that primary does go down uh a new secondary can be elected and take its place within second so it happens pretty quickly you're not talking about massive amounts of downtime in the event of a primary failure but you do need to make sure you get that primary back up online pretty quickly because if your Operation Log Runs Out of Space during the time that it's been down recovering that primary is going to get a whole lot more difficult so you know you need to make sure that you're still you still have some operational responsibilities to actually get that back up and running quickly and I want to stress again that we haven't even talked about Big Data yet what we're talking about here in replica sets is just having a single monolithic mongodb server where all of the data sits on that single server and we're replicating that data to backup servers okay so we're not talking about Big Data yet we're just talking about durability and actually having backup copies of a single monolithic mongodb database here there are a lot of quirks with mongodb and it's uh you know something that it does gets its share of criticism for one thing is that you have to have a majority of servers in your set to agree on who the primary is so you can't have an even number of servers because you can't get a majority and that implies that you need to have at least three servers if you want to have replication or some sort of durability and that can get expensive right maybe it doesn't make sense to actually have three giant servers just to keep your one M mongodb instance reliable so to get around that limitation they have something called an Arbiter node that you can set up in the place of a secondary node where its only job is to vote on who the primary should be in the event of a failure so that's an option but you can only have one Arbiter node in your cluster so I don't know it's a little bit weird the other thing is that your applications need to know about at least a few servers in your mongod DB cluster so it needs to know about you know your current primary and a few secondaries at least so it can actually ask one server who the primary is that it should be talking to so that means that if you're going to be changing the configuration of your servers or adding more secondaries or removing secondaries at the end of the day you need to push that information all the way up to your applications which it can be kind of a pain and again I want to stress that replica sets only address durability we haven't talked about scaling out to Big Data yet um if you're if your replica set goes down for whatever reason your your database is down okay so there is a way to set things up so that you can read from secondaries but generally that's not recommended so we're just talking about durability here another NE but one neat thing about replica sets is that you can set up something called a delayed secondary and the idea there is that you can set up a time delay between the replication between your primary and a specific secondary node and you can do that as insurance against doing something stupid so for example let's say I set up a 1H hour delay between primary and secondary replication and I do something really dumb like accidentally drop an entire database on my mongodb instance if I can catch that quickly enough I can shut things down and restore from that delayed secondary to get back to where I was an hour ago and restore that information relatively quickly let's talk about Big Data that's why we're here so for actually scaling out data across more than one server with mongodb we need to set up something called sharding and the way sharding works is that we actually have multiple replica sets where each replica set is responsible for some range of values on some indexed value in my database so this in order to get sharding to work it requires that you set up an index on some unique value on your collection and that index is used to actually balance the load of information among multiple replica sets and then on each application server whatever you're using to talk to mongodb you will run a process called s and S talks to exactly three configuration servers that you have running somewhere that knows about how things are petitioned and then uses that to figure out which replica set do I talk to to get the information that I want so let's take a look take a minute to understand this architecture here we can have many application servers these might be web servers on some big web app for example where each process of your web servers is running an instance of mongoos s has some communication with three configuration servers you're running somewhere and these can run on on top of other servers you might have they don't have to do a whole lot of work but you need to have three of them and from there I can figure out which replica set to talk to to actually read or write the information for a given um say user ID or something that you're indexing on and that replica set in turn can take care of durability and actually backing that data for that replicat replica set up to a bunch of secondary nodes that it can fail over to now s is running something called a balancer in the background so over time if it finds that it's actually doesn't have an even distribution of values in whatever field you're partitioning on it can rebalance things across your replica sets in real time over time so in this example we might have replica set one that's set up to handle user IDs you know from the minimum value to user ID 1000 maybe replica set two is handling user IDs 1,000 to 5,000 and replic replica set 3 might be handling user IDs 5,000 to whatever the maximum value is so these can change over time and get rebalanced over time as the need arises so that is how mongodb handles Big Data you can see it's actually pretty complicated uh but you know if you actually to be fair if you compare this to something like hbas where you're using something like zookeeper to maintain these sorts of conf configuration it's not that different charting itself has some quirks in mongodb so for example Auto sharding where it's trying to rebalance thing over time sometimes fails there is a rather nasty failure Mode called a split storm where it simply cannot split things quickly enough and it just keeps trying to resplit things over and over over and over again and your entire cluster goes down it's a bad thing another failure mode is if your s processes on the front end get restarted too often things will never rebalance so it actually takes a look on each s process over time to see how data is being distributed throughout your cluster and if you keep restarting it it basically restarts the clock restarts the count on those things so if you are restarting those processes too often and sometimes depending on how you set up your web server that might be pretty that might be the case things won't won't be balanced properly so it's very easy to get into a bad State got to got to make sure someone's really keeping eye keeping an eye on things from an administrative standpoint you do need to have exactly three config servers and if any one of them goes down your entire database goes down and this really isn't any different from hbase where you have you know Master nodes that are maintained via zookeeper so again we're trading off intentionally um consistency for availability and the other thing too is like I said before even though mongod DB offers a very loose defined document model it doesn't mean that your document model should be loose if you're going to be doing sharding and actually handling Big Data you still need to think about having some single primary key that is unique to each document that you're going to be charting on now I've kind of talked a lot about the uh limitations of mongodb but there are some very neat things about it too so again you know the the big plus of mongodb is that it's not just a nosql database but it can store pretty much anything you want it also has a shell that has a full JavaScript interp so there's a lot of power there that you can do you can actually run JavaScript functions across your entire mongod DB database pretty easily it also supports many indices although you're still discouraged from doing more than two or three in a given collection and you can only have one that's used for sharding but you can actually set up things like full text indices for doing efficient text searches across mongodb so again mongodb is really a good choice for things like storing you know big documents of information or text you can also have spatial indices where you can have actually do searches across you know latitudes and longitudes for example and try to figure out what database objects actually intersect a given position for example which is kind of a neat feature another thing about mongodb that's worth talking about is that they're kind of trying to make mongodb into a replacement for Hadoop to some extent so you it actually has built-in aggregation capabilities you can actually run map reduce code on mongodb itself and it actually has its own file system built in as well called grid FS that's kind of like hdf in some extent where it's storing documents within mongodb and actually chunking those documents up kind of like hdfs does so mongod DB's kind of value proposition is in part the fact that for many applications you might not need Hadoop at all mongod DB might be all that you need but if you are integrating mongod DB with Hadoop or spark or something like that it's easy to do as we'll see in a moment and the good thing is that it can actually leverage some of these features in mongod DB to do things more efficiently so for example if we're tying mongod DB to a spark data set and you're telling spark to go perform some map redy task on mongodb that work might actually get pushed down to mongod DB itself so it might not not actually have to use to dup at all that can actually lead to you know more efficient data analysis and you might be able to get from other nosql solutions that are integrated with something like Hader spark and there is actually a SQL connector available for mongodb so you can actually write full-blown sequel against it if you want to but bear in mind it's still not really a relational database even if you have the ability of executing SQL commands against it you still can't do efficient joins and can't deal with normalized data very efficiently so with that we've talked a lot let's actually go play around with mongodb let's actually look at integrating mongodb with spark and get some data into it and then we can play around with the data in mongodb and see how it works from within the shell so let's go have some fun