Joe Doyle's Coding Blog

Posts tagged "MongoDb"

A common question I've seen on StackOverflow asks for the best way to open a connection to MongoDB when starting up your Express app. Folks generally don't care for just putting all of the Express setup in the callback of the MongoDB connect call, but it seems to be the generally accepted approach. I didn't like it either and felt that there must be a better way. Here's what I came up with. The Callbacks You can't really escape the callbacks when dealing with the native MongoDB driver. Pretty much every call expects a callback. The way I deal with that is by using Promises using the Q library. Q goes beyond just providing a way to use promises by also providing helper functions for wrapping existing Node.js APIs that use the standard callback pattern of the function(err, result). Promises are a deep topic themselves, so I won't go into them in detail here. Just know they they can help turn the callback "Pyramid of Doom" or "Callback Christmas Tree" into a chained series of function calls which greatly improves the readability of your code. Google can hook you up if you want to know more. The Database object The first step that made the most sense when I first started using MongoDB in Node.js was to create my data access object. Its used for creating the connection, holding the references to the collections used, and the methods that perform the specific actions against MongoDB. So here's what my Database object looks like: var Q = require('Q'), MongoClient = require('mongodb').MongoClient, ObjectId = require('mongodb').ObjectID, Server = require('mongodb').Server, ReplSet = require('mongodb').ReplSet; _ = require('underscore'); var Database = function(server, database) { this.server = server; this.database = database; }; Database.prototype.connect = function(collections) { var self = this; var connectionString = "mongodb://" + this.server + "/" + this.database + '?replicaSet=cluster&readPreference=secondaryPreferred'; return Q.nfcall(MongoClient.connect, connectionString) .then(function(db) { _.each(collections, function(collection) { self[collection] = db.collection(collection); }); return db; }); }; Database.prototype.findDocs = function(term) { return this.mydocs.find({ Title: term }).stream(); }; Database.prototype.saveDoc = function(postData) { return Q.npost(this.mydocs, "update", [{ id: postData.id }, postData, { w: 1, upsert: true }]); }; module.exports = Database; So what's going on here? For the most part, nothing very exciting. We take in the server(s) and database we want to connect to in the constructor. The first interesting part starts on line 16. Q.nfcall is our Q helper function wrapping the MongoClient.connect and giving us a promise back. We chain a then() function which is called after the connection is made to MongoDB. The function receives the connected db object which we can then save a reference to each collection we want to use for our app. We then return the db object from the function so we can keep passing it along. The end result, which consists of the chain of our two functions, is still a promise which is then returned back to the caller. Just to show a little more detail, you can also see the Q library in use for performing an upsert when we want to save a new document. Again, the promise is returned which means we don't need to use a callback. Line 27 also shows that the find function can utilize streams instead of using a callback. I hope that feature is spread around more! The Express App Configuration For the Express configuration, I decided to keep most of it wrapped in a function. Most of it could be pulled out and just run before we initialize the database. I like that its wrapped up, personally. So here's what our app.js looks like: var database = new Database(settings.databaseServers, settings.database); function startServer(db) { app.set('port', process.env.PORT || 3000); app.set('views', __dirname + '/views'); // The rest of the setup is excluded for brevity... console.log('Connected to the database'); app.locals.database = database; routes.registerRoutes(app); http.createServer(app).listen(app.get('port'), function onServerListen(){ console.log('Express server listening on port ' + app.get('port')); }); }; database.connect(['Posts', 'Stats', 'Log']) .then(startServer); The code that really starts things off is at the bottom. We call connect on our Database, passing in the array of collections we want. Since connect returns a promise, we can tack on another function using then() which will also use our connected db object. In this case, it's our startServer function which loads up Express and start our server listening. Accessing the Database in your Routes In our app.js snippet, something I do is attach the database to app.locals on line 10. I'm not sure if this is the best approach, but it has been working for me so far. Now in my routes, I can access the database using req.app.locals.database. It could also be passed in to the registerRoutes function and pass around from there. For my blog, instead of accessing the database directly from the req.app.locals reference, I have another layer which resembles the Repository pattern. For simpler apps I've been ok with the direct reference approach. Can it be better? Like most of the code we write, it looks pretty good today. Much better than how we did it last year. I'm not sure if there's a better way, with better falling into [simpler, more scalable, something I just don't know about]. If you know of or use a better approach, I'd love to hear about it!

A Pattern for Connecting to MongoDB in an Express App

A common question I've seen on StackOverflow asks for the best way to open a connection to MongoDB when starting up your Express app. Folks generally don't care for just putting all of the Express setup in the callback of the MongoDB connect call, but it seems to be the
A few years ago I wrote about using the MongoDB driver in C#. Its been one of my most popular posts and it really needs an update! Since 2011, the 10gen driver has become the standard. Its been getting updated on a regular basis with features that bring it inline with what we would expect in C#. I've been using MongoDB for all of my personal projects and have been very happy with it. So here's an update for what it looks like today to use the MongoDB driver version 1.8.1. Getting the Driver The source is still located on GitHub at https://github.com/mongodb/mongo-csharp-driver. But now that we have NuGet, the easiest way to get started is by using the NuGet package mongocsharpdriver (http://nuget.org/packages/mongocsharpdriver/). It's everything we need compiled and ready to go. To the Code The best place to get started is still the official 10gen C# driver tutorial at http://www.mongodb.org/display/DOCS/CSharp+Driver+Tutorial. It covers what you need to get started and helps to keep track of what is new each release. I'm going to stick with the original app which was a simple tool that keeps track of passwords and various notes. I've been using it for a few years and have been happy with it's simplicity. The first change is how we connect. It used to look like this: MongoServer server = MongoServer.Create("mongodb://myserver"); MongoDatabase db = server.GetDatabase("TheDatabase"); That style has been deprecated and instead we should use the new MongoClient class. Here's what that looks like now: var mongoClient = new MongoClient("mongodb://myserver"); mongoServer = mongoClient.GetServer(); var database = mongoServer.GetDatabase("TheDatabase"); Not too different, but the naming is definitely clearer. It no longer looks like we're creating a server which is nice. Getting a reference to the collection is still the same. We specify our CredentialSet class when we get our collection so that we don't need to work with BsonDocuments if we don't want to. Even though MongoDB is a schema-less document store, it does make life easier to have a fixed type to work with. var passwords = database.GetCollection<CredentialSet>("passwords"); And just as a reminder, our model looks like this: public class CredentialSet { public ObjectId Id { get; set; } public string Title { get; set; } public string Username { get; set; } public string Password { get; set; } public string WebSite { get; set; } public string Notes { get; set; } public int Owner { get; set; } public DateTime LastUpdate { get; set; } } One thing that did get fixed is the _id issue from last time. The driver will now use Id as the document id automatically. It also will look for _id, but that's not inline with C# standards. So lets save a new document: var password = new CredentialSet(); // set the property values. passwords.Save(password); Now that we have a saved document, let's query for it. Here's where we get to my favorite new feature; support for Linq. Instead of building up a Query object and using Find or FindAll, we can access our collection as an IQueryable and use Linq against it. Like most custom Linq providers, not every operation is supported, but its typically nothing that can't be worked around. So before we had: var query = Query.EQ("Title", "A password"); var oneDocument = passwords.FindOne(query); Now we can do: var result = passwords.AsQueryable().SingleOrDefault(x => x.Title=="A password"); Of course the older methods are still available for the operations that don't make sense with Linq, such as map/reduce. Next Time: Aggregation Framework, V8, and Full Text Search Another nice feature in MongoDB 2.1 and later is the Aggregation Framework. It provides an easier alternative to map/reduce. I'm still learning about it, but I am using it on this site to generate some statistics for my dashboard view. As of 2.4, V8 now powers MongoDB and we get a few extra benefits such as multiple scripts executing at the same time. We also got the first version of a full text search engine built into MongoDB. I'll dive into these next time.

Getting Started With MongoDB and C# Revisited

A few years ago I wrote about using the MongoDB driver in C#. Its been one of my most popular posts and it really needs an update! Since 2011, the 10gen driver has become the standard. Its been getting updated on a regular basis with features that bring it inline
The World of WordPress About a year and a half ago I made the switch from Posterous to a WordPress for my blog. I figured that I might as well learn how to use the 800 pound gorilla in the room. For the most part, things went well considering that I wanted to run it on Windows under IIS and use SQL Server as the database. I added some plugins for the basics like commenting, syntax highlighting and the like. The import from Posterous was smooth with nothing lost. And it was good. I'm not sure if it was the WordPress upgrade or an update to one of the plugins. One day a few months ago I tried to create a new post only to have 90% of it just disappear upon hitting save. I hit the edit button and retyped a paragraph to see if that would save. It didn't. Typed a little less then previewed the post this time. Gone. I did a few more experiments with creating new posts and editing them in various stages. They all seemed to auto-save early in the entry and then get locked forever. A Ghost in the Darkness I wasn't sure what I wanted to do about my blog. I wasn't in the mood to re-install WordPress. I looked at a few blogs written in .NET, but none of them really appealed to me since most are written using WebForms. Then I saw the KickStarter for Ghost popup on Twitter. It's basically the start of a new platform designed to focus on blogs vs the CMS style product that WordPress has become. Its written in Node.js with SQLite as the default backend database. Markdown is used as the input language with a real-time preview as you create a post. It looks to leverage the best of HTML5 to make a state of the art blogging platform. My initial reaction was probably the same as most developers when they see something cool on the web: I can build that! And so I did. "I see you have constructed a new lightsaber." There's a bit of me that feels writing your own blog is a rite of passage as a developer. I know most people use existing packages because why would you really want to waste time at creating something that has been created hundreds of times before. For me, this is a chance to not only give it my personal touch, but really experiment with new technologies and practice the skills outside of my comfort zone. Some might say it's like a Jedi building his first lightsaber. At work I almost exclusively use ASP.NET MVC 4. And while I really do like using it, I felt this was the perfect time to try building a website in Node.js and Express. I really liked the idea of using Markdown instead of a WYSIWYG editor or plain HTML. I also liked the idea of having the layout update in real time when writing a post. I'm using MongoDB since it's my go-to datastore due to how easy and fast it is. So far the core is done. It's still mostly MVF (minimum viable functionality), but I'll keep tweaking it as I go. Here are some of the highlights that I'm proud of or really happy with. Editing To get the dual Markdown/HTML rendering I'm using Pagedown which is from the folks at Stack Exchange. Its the editor they use on their sites. It was really easy to implement and there's even a 3rd party add (Pagedown.Extra) on which extends the Markdown a bit more for things such as tables and code syntax highlighting. For syntax highlighting I'm using SyntaxHighlighter. For uploading images and files I integrated Dropzone.js by overriding the image dialog in Pagedown. Dropzone is amazingly simple implement and provides thumbnails of the images as you upload. Just eye candy, I know, but the effect is sweet. Here's a screenshot of me writing this post: Styling If there's anything I need more practice at, it's design. Thanks to Twitter Bootstrap, I got a running start. I like the clean and simple look so I tried to keep plenty of whitespace and let it define the sections. I'm using LESS for the CSS. I'm not yet customizing Bootstrap, but its on the list. Font Awesome is used for the icons. I went pretty minimalistic on the colors sticking to the really-dark-grey-and-black on while. I'm still iterating over the layouts, but I think I'm pretty close. Hosting I run my own servers, so I wanted continue to host my blog locally. For now I'm using iisnode with Node.js 0.10. One of the benefits is that I can have IIS host all of the static content and only have Node host the dynamic pages. This is the standard Node configuration I hear about, with the exception that its Nginx used instead of IIS. The concept is the same. I have Grunt setup to do my build and deployment so I can test locally then push out the live site. I really like Grunt and am looking at how feasible it would be to use in the .NET world for things like project scaffolding. Performance I wanted the site to be fast. Really fast. I tried to do all that I could to optimize the site. Grunt combines and minifies my JavaScript and CSS. Express is gzipping the content. The slowest part of the site is Disqus which is used for comments. Without Disqus, page load times are sub-70ms. Someone said on Twitter that a blog without comments is not a blog (and I agree), so its a price I'm willing to pay. One way I make things fast is loading all posts in memory and keeping them there. I don't have thousands of posts, so I can get away with that. Right now Node is only using ~60MB of memory, so I'm not too concerned. Almost there I still have a few behind the scenes sections to create. I want to build up a dashboard for some stats. Probably won't be as amazing as what Ghost will provide, but I'm not sure I need that much. I still have Google Analytics running anyways, and its not like I'm going to beat that. I also want to pretty up the Edit page to use auto-completion for the tags and to have the url get built from the title automatically. Just a bit of extra polish really. I do have an RSS feed, so if you're interested in .NET and Javascript posts, please do subscribe. Until next time...

The Creation of My New Blog

The World of WordPress About a year and a half ago I made the switch from Posterous to a WordPress for my blog. I figured that I might as well learn how to use the 800 pound gorilla in the room. For the most part, things went well considering that
While most of this article still applies to the current version of the C# driver, I have written an updated version: Getting Started With MongoDB and C# Revisited My main goals are to setup MongoDB for small scale applications that aren’t going to scale up to lots of users and multiple servers. I’ve installed MongoDB as a service and I’ve started to play around with the 10gen C# driver. There are a couple of C# drivers already out there (NoRM and the mongodb-csharp ones being the most popular) and people report varying levels of success. 10gen has also released a driver which has caught up feature-wise to the others. I decided to use this one because it’s easy to use and I do get a warm and fuzzy feeling knowing that it is from the 10gen folks. You can find the repository at https://github.com/mongodb/mongo-csharp-driver Into the code! The best place to get started is the 10gen C# driver tutorial at http://www.mongodb.org/display/DOCS/CSharp+Driver+Tutorial. It covers what you need to get started and sometimes a bit more. The app I’m writing is a simple one that stores usernames and passwords along with some other information like the URL of the website to use the password and any notes that we might want to add. It’s just going to be on our internal network and won’t have any interface to the internet. That means I’m not focusing on things like encrypting the passwords, or other security measures if this were to be used anywhere else in the public. Connecting to your database is pretty straight forward. There are many other options available, but just to get started, you just need the server name. MongoServer server = MongoServer.Create("mongodb://myserver"); MongoDatabase db = server.GetDatabase("TheDatabase"); One of the things I like about MongoDB is that just asking for the database will create it. So now that we have a reference to our new database, what are we going to do with it? Most of the time, we already have a model ready to be stored. Here’s my class that hold a password set. public class CredentialSet { public ObjectId _id { get; set; } public string Title { get; set; } public string Username { get; set; } public string Password { get; set; } public string WebSite { get; set; } public string Notes { get; set; } public int Owner { get; set; } public DateTime LastUpdate { get; set; } } It’s a pretty basic class. The only addition from standard C# is the ObjectId class. This class represents the default MongoDB identifier. You can choose to use your own unique identifier, but for now, I’m just going to use the default. Our next step is to create an instance of our class and save it to the database. But first, we need a place to store it. In the relational world, we would use tables inside our database to store the data. In the MongoDB world, we use the Collection. Just like the database, the act of getting the reference to it will create it if it doesn’t exist. MongoCollection passwords = db.GetCollection("passwords"); As you can see, we can specify our CredentialSet class when we get our collection. Even though MongoDB is a schema-less document store, it does make life easier to have a standard, static type to work with. When we specify a class like this, we are telling the driver to use our CredentialSet as the default when pulling our documents from the database. You can still insert any type of document you want, but this style saves us some key strokes later on. So now let’s save our document. var password = new CredentialSet(); // set the property values. passwords.Save(password); We can now use a tool like MongoVUE to see our record in MongoDB. When we take a look at it, we see something a little unexpected. Our _id is all zeros! /* 0 */ { "_id": "000000000000000000000000", "Title": "A password", "Username": "username", "Password": "password", "WebSite": "www.google.com", "Notes": "This is a password!", "Owner": "1", "LastUpdate": "Tue, 1 Feb 2011 10:47:20 GMT -05:00" } Doing some research, I found some references to a known issue with the 10gen client, and a simple fix. We just need to add an attribute to our model’s _id property. Here’s the updated CredentialSet. public class CredentialSet { [BsonId] public ObjectId _id { get; set; } public string Title { get; set; } public string Username { get; set; } public string Password { get; set; } public string WebSite { get; set; } public string Notes { get; set; } public int Owner { get; set; } public DateTime LastUpdate { get; set; } } This tells the driver that we want to use the _id property as the internal MongoDB identifier. After delete our exiting item using MongoVUE, we can run our sample again and examine the record. /* 0 */ { "_id": "4d38833880844214f0a8c60b", "Title": "A password", "Username": "username", "Password": "password", "WebSite": "www.google.com", "Notes": "This is a password!", "Owner": "1", "LastUpdate": "Tue, 1 Feb 2011 10:47:20 GMT -05:00" } Much better. Now we can try to pull that document out. There are lots of queries you might want to do. Way more than I can go through. I’m just going to show two simple examples. The first is pulling out all documents, and the second is finding a specific record based on a single field. Let’s start with all records. var allPasswords = passwords.FindAll(); It doesn’t get much easier than this! Again, we can use this simple method because we’ve specified a default document class. From here, we have a collection of CredentialSet objects that we can work with using standard methods such as foreach or Linq to Objects. So now let’s get a specific document. To get a specific document, we need to build up a Query object to tell the driver how to create the JSON that MongoDB will use to find our document. From there, we use the FindOne method on the collection. var query = Query.EQ("Title", "A password"); var oneDocument = passwords.FindOne(query); There are lots of options when creating a Query. The one we used here, EQ, does a simple Equality comparison. It finds all documents where the Title field exactly matches ‘A password’. Since this was the Title we put in above, that’s the one we get back. Just about all of the options for querying are available. The 10gen C# driver page does a good job covering them. Wrapping up This was my first use of MongoDB. With the basics of saving and retrieval down, I can move forward on getting an app up and running. I know that this is really simple and it doesn’t cover any of the features MongoDB is known for such as master/slave replication or sharding. I also don’t do any error checking. Something as simple as this can be done with any relational database. But in order to do this, I’d need to hook in an ORM such as nHibernate or EF4. That means extra code. The MongoDB driver handles all of the class to JSON mapping automatically. That’s what I’m looking for with this. Standard tutorial disclaimer: None of this code is what I consider Production Ready. It did give me a starting point to move forward from. Hopefully it helps someone else as well.

Getting Started With MongoDB and the 10gen C# Driver

While most of this article still applies to the current version of the C# driver, I have written an updated version: Getting Started With MongoDB and C# Revisited My main goals are to setup MongoDB for small scale applications that aren’t going to scale up to lots of users
I’ve been reading up on MongoDB. I picked up MongoDB: The Definitive Guide. It’s not a bad reference to have. I’m actually surprised how thin it is, yet it answered just about every question I had. It covered everything I wanted to know in getting started with MongoDB when looking at it from a “SQL” perspective. It did a good job explaining about de-normalizing your schema because document databases just don’t work that way. It also covered the basics of accessing subdocuments, or arrays in arrays. I pretty much understand its benefits and limits as a database. Rightfully so, it didn't really cover single instance scenarios to the depth I have been thinking about. Sure, we all know how great all of the NoSQL databases scale when you need to handle 1.21 bajillion requests at once. But what if you want to use the easy and speed of storing and accessing data for a small app? I know it’s not sexy enough to cover in the main stream, but what about the people that want a document database for use in a small environment? That’s what I want to know about. What if I want to create the back end of a billing system or CRM that isn’t going to scale beyond a single office? What if I’m never going to go past a thousand users? I think that there can be a great benefit to the small app developer by using a document or key/value database instead of only a traditional ACID compliant, relational database. I’m going to experiment with using MongoDB as a central database for a few small apps. Why shouldn't we use the ease of access provided by these systems for apps that will never hit millions of users? Sure, ORMs are great, but what if we never had to use them at all? My next little hobby project will be to convert a little password app for my wife to sync up to a MongoDB server. It currently uses a local SQLCE database which works just fine. I think by expanding it to also save on one of my servers, it will provide a backup of the data in case her laptop crashes. It will also let her search my passwords, and me, hers. I want explore the best settings for MongoDB when you only plan on using it on a single server. I feel that this is an under served area. There are lots of companies, for better or for worse, that don’t have the ability to scale out across multiple servers for their critical systems. Should they stick with traditional relational databases or can they too enjoy the performance benefits of a document database?’ The goal for my next post is to provide the answers to this question. What is the best configuration for a single server environment?

MongoDB and scaling down?

I’ve been reading up on MongoDB. I picked up MongoDB: The Definitive Guide. It’s not a bad reference to have. I’m actually surprised how thin it is, yet it answered just about every question I had. It covered everything I wanted to know in getting started with