Well, there are the obvious reasons like length, age, unpopular subject matter, or sometimes just that the opportunity cost would be too high. Then there are less talked about reasons like “That manga-ka is a pain to work with” or “That manga-ka won’t allow us to license their work” or “We think that series is terrible, actually” or “There are licensing issues that make publishing that here impossible.” Oh, but I rarely talk about those latter issues, for obvious reasons. I do not want to potentially insult or anger the Japanese authors. ^_^;
Length: the longer a series gets, the more readers it’ll lose. The rule is that more people will buy volume 1 than volume 2, more people will buy volume 2 than volume 3, etc. This is true for even fabulously successful series— reader attrition is just something you have to know about and plan for from the beginning. If we don’t think the audience for a particular title will be large enough at the start to support the series through its entire run, we probably won’t license it.
Age: In addition to older manga not selling very well on the whole, we also have to consider the additional time and costs it adds for us to publish older material. Nowadays, almost all of the pages we get from Japan are clean digital files— with all of the Japanese text already removed (sometimes even the sound effects too!). This is MUCH easier to work with on our end than older scans, where we have to retouch the text out of each page (like we did for Sailor Moon, and are currently doing for Tokyo Mew Mew a la Mode, actually). And that is assuming that the manga-ka even still HAS the older files in workable condition— sometimes they don’t. So when the choice comes down to a newer series or an older series, we’re much more likely to go for the newer one, since that’s where the biggest chance of a payoff with the least cost to us lies.
Unpopular subject matter: So there are two aspects to this: things that are unpopular with bookstores and things that are unpopular with consumers.
Bookstores are still where we sell the bulk of our books, so if the content of the series is something that we know they wouldn’t like (if it’s extremely sexually explicit, say), we’d probably rather skip it. This can also come back to bite us if something by that author has been published here before— but it bombed terribly. Even though we can swear it’ll be different this time, the bookstores will be less enthused to stock that series since they view that author as unprofitable. So if a previous series by any given manga-ka is a dud, we’re unlikely to look into more work by that manga-ka. (and the reverse is true! If a series is popular, we generally start considering their other series too. See: Hiro Mashima.)
Then there are certain subjects that we know are unpopular with consumers after years of industry trial and error— sports series, 4-koma manga, josei manga, gag comedies filled with Japanese minutiae, etc… to name a few, but you get the picture. Of course there are always exceptions, but betting that your chosen license is somehow an exception to the norm is not very wise.
Opportunity costs: we can only publish so many series per year with the resources we have, so sometimes series get pushed aside not because they’re not good too, but because we had to make decisions somewhere and went with other series that time. Sometimes series that get passed over like that will make it in the next time we discuss licenses.
…so, there you go! Those are a lot of the most common reasons why we might choose not to license a particular series.