Merging M4V files on a Mac ... with chapters!

As I’ve mentioned a couple of times before, one of my projects right now is ripping all the DVDs I own so that I can watch them on my AppleTV (or any AppleTV in the house). Well, one of the problems I’ve run into a couple of times is longer movies that are distributed on two discs. This is usually movies like the Lord of the Rings Extended Edition or The Ten Commandments. Really, they’re one movie, but are distributed as two separate movies because of the restrictions of physical media. Well, digital media imposes no such restrictions on us, so why have two separate movies listed on the AppleTV? So after much trial and error, I finally discovered a way to get everything play nicely together. Unfortunately, this is not an easy problem to solve and even involved me writing a small script that could merge chapter files together because every single method I could find would eliminate chapter markers. So here, in abbreviated form, is the process for merging m4v files together and preserving chapter markers. Note: This tutorial assumes some level of technical proficiency. This is not a point-and-click process (yet :P) and requires the use of multiple tools and the shell. Tools you’ll need: Handbrake or whatever tool you’re using for ripping your legally obtained DVDs. MetaX and/or iDentify Subler remux Quicktime, which is now built into Mac OS X. chaptermerge, a script I wrote that merges chapter files together. The proces: Rip both movies from their individual DVDs using Handbrake or whatever other tool you’re using. Be sure that you’re adding chapter markers. Load each movie into MetaX and download the chapter names. That’s really the only thing you need to add to the file. Save the files with chapter names. Load each movie into Subler and extract the chapter files. To do this, select the chapter track and select File -> Export. Now, open the first movie in Quicktime. Drag the second movie on top of the first one. Quicktime will add the two together. Save the movie for use on an AppleTV. Get a beer or 6, because this takes awhile. While the movie is saving, use chaptermerge to merge the chapter files together. See the docs on how it works. Once the file has finished saving as a Quicktime MOV (it’s actually still h.264 inside the file), fire up remux and convert the merged file back into an m4v. Drag the file into remux, set the output to m4v, and save. Should be pretty quick - a matter of minutes. Load the merged file back into Subler and add the merged chapter track. Drag the chapter file into the Subler window. Save the file. Load the merged file into a tool such as iDentify or MetaX and add the remaining metadata. That’s it! You now have a merged file with both parts of the movie, accurate chapter markers and full metadata, ready to be copied to iTunes and viewed on your AppleTV.

A Year With Drobo: My Review of the Drobo FS

About a year ago, I picked up a Drobo FS. It was something I had been wanting to do for awhile to support my ever growing data needs. In particular, I had three problems I was aiming to solve: Data security. In addition to the obvious suspects of photos and home movies, I have a lot of old files and documents I’ve been hanging onto for  years now. Papers I wrote in high school and college, some of the first computer code I wrote, etc. How I’ve managed to preserve some of this over the years is a miracle in itself - a lot of it was recovered a few years back when I picked up a 3.5” floppy disk drive and started going through boxes of floppies in my attic. But now that I have it all in one central place, I’d like to secure it. Media library. My wife and I own a lot of DVDs, and they take up a lot of space. They’re also not very portable. A while back I started the process of ripping all my DVDs into iTunes so that they could feed to any TV in the house with an AppleTV, essentially creating our own private video on demand system. This was rapidly outpacing the available space. Central backup location. I wanted a place where all the Macs in the house could backup to via Time Machine. After doing a great deal of research, I decided on the Drobo FS. In addition to being able to do all of the above, it had some other nice features that I liked: Thin provisioned, meaning you can hot-swap drives in and out while the device is running and not have downtime while it rebuilds the array. Also thanks to thin provisioning, your drives (theoretically, I’ll get to this in a bit) don’t need to be the same size or from the same manufacturer. Data protection that purports to examine the health of a drive and move data around to give it the best change of preservation. Now, to be sure, you’re trusting a black box. If Drobo fails, there is almost no cheap way to recover that data as they use non-standard, proprietary technology to accomplish all their voodoo magic. Nonetheless, in this case, it was a tradeoff I was willing to make. So after a year of ownership, how does Drobo stand up in fulfilling these promises? Well, there’s a lot of stories here but, overall, it does well with a few caveats I’ve learned along the way. When I first ordered the Drobo, at the same time I placed an order for 3 identical Seagate 2TB drives. I got them installed and got the array up and operational, and got all my data from various places moved over to the Drobo. The first thing I noticed while copying data over to the Drobo was that it was slow. Very slow. Transfer speeds to the Drobo across my gigabit network were in the ~10 megabit range. Upping the frame size to jumbo (9000) improved that a little but it was still very slow. Not a deal-breaker, as you’re rarely moving that much data around, but it was something I noticed. Then, the real problems started. The Drobo would just randomly vanish from Finder. No reason, just one moment it wouldn’t be there and you couldn’t even connect to it via IP address, although you could still ping it. I opened a support ticket with Data Robotics, who took me through a troubleshooting procedure that involved directly connecting the Drobo to my Mac via Ethernet. Of course it would work fine when we did that, so I figured it was a problem with my network. But even creating the shortest possible path between my iMac and the Drobo yielded the same results. I opened another ticket, and we went through the same procedure again. This time, however, we let it sit longer. Sure enough, about 30 seconds after it booted and appeared in Finder, it disappeared from Finder and from their Dashboard tool. I was able to SSH to it and see that the filesystem was now mounted in read-only mode. But we were able to get some diagnostic log files off of it. The tech looked them over and said that the drives were failing. And sure enough, the next day, Drobo reported one of the drives had died and that it was moving data around to protect things. Now, in my entire life, I’ve had 2 hard drive failures, with one occurring just a couple years ago. So I popped online and ordered another drive (this time, a Western Digital Enterprise 2TB drive). Popped it in the Drobo and it seemed happy, although still slow and occasionally vanishing from finder. Then, about a month later, I’m out working in the yard and boom, get an email on my iPhone about a second drive failure in the Drobo. So I order another Western Digital 2TB drive and put it in. The whole time, by the way, Drobo remains on and accessible. Pretty cool actually. And replacing a drive is pretty easy - you just pop the old one out and put the new one in, without even shutting down. Drobo then goes into a “protection” mode where it shuffles data around onto the new drive. But, with 2 of the 3 Seagate drives I bought failing within 6 months, I decided it probably wasn’t wise to continue to trust that last one since it was probably from the same batch. So I replaced that one as well. That was about seven months ago, which brings us to today. Between the better drives and several firmware and software upgrades from Data Robotics in the interim, Drobo is now virtually rock solid and (knock on wood) I haven’t had any further problems. It no longer randomly disappears from Finder or the Drobo Dashboard and, in a very Apple way, it just works. And I also want to say that, throughout the troubleshooting process, the Data Robotics guys were great to work with and wanted to see the problem solved. So overall, after a year and some growing pains, I’m pretty happy with it and wouldn’t hesitate to recommend the Drobo FS with the following caveats: Use quality drives. Don’t buy the cheap drives and definitely avoid Seagate drives as the Drobo seems to hate those. When I upgraded to the WD drives, I bought the server-level Enterprise drives. Those have been rock solid. My guess is that Drobo is pretty hard on drives, with lots of reads, writes and seeks. Use the same size and manufacturer. Now, one of Drobo’s big selling points is that you can use different size drives and all that. This is one of those cases where what you can do and what you should do are two different things. You can use any size and manufacturer, but I’ve had better success and performance when all my drives are from the same manufacturer and are the same size. Be sure your firmware and software are up to date.  Kinda goes without saying, but the firmware upgrades for the Drobo have really helped with it’s stability. If you start having problems with your Drobo like I had above, get ready for a drive failure.

Restoring a Mac from a Time Machine backup on a Drobo (or other network storage)

Been having some problems with my iMac upstairs. I’m pretty sure the hard drive is failing (again), although hopefully it’s just bad sectors. But, with hard drive prices currently still in the stratosphere, I decided to try one more last trick to see if I can save myself some money. That is, the old Windows trick: fdisk, format, reinstall. Or, well, the Mac Equivalent - Disk Utility, reinstall. About a year ago, a bought a Drobo. I’ve been meaning to write a review of the Drobo and maybe now I will (the short of it is, I had some growing pains with it, but now that I’ve figured out its quirks, it seems to work well). One of the reasons I bought the Drobo as to use a shared Time Machine backup store for all the Macs in the house. So, I thought, in addition to trying to save my Mac, now would be a great time to test my fancy Time Machine backup system. And, unfortunately, since Time Machine really isn’t meant to work with unsupported network volumes, it does require some gymnastics to get it to work. Even worse, it isn’t a very well documented procedure. But, ultimately, I was able to figure it out; I’ll post what I did hoping that maybe it will save someone some time and headache. First step is to format and reinstall as you normally would. If you are on/installing Lion, you may be presented with an option to reinstall from a backup as part of the install process. Don’t do this. Reinstall Lion as if you were performing a fresh install. When the installation is complete and you get to the Lion post-install setup screens, you will (eventually) reach a screen asking you to create a user account. Create your original user account (same username) as in your backup. Once you’re out of setup, go to System Preferences, then Users. Create a new administrative level user (I called mine “foo”). Be sure this is an admin-level user. Log out and log into the account you just created. Turn on unsupported Time Machine volumes. Open up a Terminal window and enter: defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1 Now, open up a finder window and navigate to your Drobo or other device and to whatever network share you have your Time Machine backups stored on. Mount it. Now, inside the share, mount the .sparsebundle that is your restore image (it should be the machine name). If you open it, you should be able to see a folder called “Backups.backupdb” in it. Next, fire up the Migration Assistant. Select “From another Mac, PC, Time Machine backup, or other disk.” Hit continue. Select “From a Time Machine backup or other disk.” Hit continue. It may take a second, but, eventually, you should see a drive image and the name of your old hard drive (usually “Macintosh HD”) appear. Click continue. It may take awhile for it to parse the image. My backup image was about 350gb, and it took about 20 minutes to parse out all the information. Select what you do or don’t want and click continue. You should be presented with a dialog stating that a username on the system is the same as one in the backup. Select “Replace…” Click continue. Wait. It will take awhile. It took about 5 hours for me to do a complete restore from a backup on the Drobo to my iMac. And that’s it. Once it finished and you reboot, your Mac should be just as it was during the last backup.

The Right Way to Create an iCloud-enabled Mac App in Xcode

Because I’ve encountered this problem twice, I’m going to do a little write-up about it. As much for me as for the next person who encounters this problem. In a very un-Apple way, this process is very poorly documented and very un-intuitive from a user-developer standpoint. Everything that’s here, I’ve culled from Googling about aimlessly and finding on Stack Overflow. **Symptom: **You create a new app in Xcode with no changes and launch it. It launches just fine. You then go to the target summary settings and click “Enable Entitlements” and have an iCloud key/value store and or containers. Now you launch it and nothing happens. Nothing appears, but Xcode still thinks the app is running. **What’s Happening: **To understand what is happening, you have to go have a look in the Console application (note, the actual system Console.app, not the debug console in Xcode). Open that up and select “All Messages”. Look for something that looks like this: 1/28/12 7:49:03.945 PM taskgated: killed <your app ID>[pid 43838] because its use of the com.apple.developer.ubiquity-container-identifiers entitlement is not allowed What’s happening is that taskgated is killing your app because it’s not properly signed to use iCloud. And for some reason that is not entirely clear to me, the app being killed is not at all reported back to Xcode - Xcode thinks the app is running. So you just sit there waiting for something to happen with no clue that this sinister lurking background process has killed your app. How to fix it: There are two ways you can go from here to fix this. The first and easiest, if you are just turning on entitlements and aren’t intending to use iCloud, you can just remove the iCloud Key/Value Store and iCloud containers from the target summary. After doing this, it should work. But, if you are making an iCloud-enabled app, there’s a long list of things you need to do. First, understand that you need to be a paid member of Apple Developer Program. Log into ADC. Go to the Mac Dev Center, and the Developer Certificate Utility. Create an App ID by going to App IDs and clicking the Create App ID button in the upper right. Enter the name of your app and the bundle identifier. It usually looks something like “com.company.app”. Click Continue. Your app ID should be entered. Click the App ID you just entered, then click “Enable for iCloud.” Click save. Next, go to Certificates. If you haven’t created any certificates yet, click “Create Certificate” in the upper right and follow the directions. Note, you need both a development and an application certificate. Next, go to Systems. Be sure you’ve added your Mac (and, for good measure, any others you’ll use for development). Finally, go to Profiles. 1. Click Create Profile in the upper right. 2. Select "Development Provisioning Profile" 3. Give it a name. 4. Select the app you created in step 3. 5. Select the certificate you want to use. 6. Select the systems you want to use (I did all). 7. Click "Generate" It may take a few seconds, then it will give you a download. 8. Open the downloaded profile. It will open in the "profiles" preference pane (which doesn't seem to appear until you try to install a profile). Click install. Now, in Xcode: 1. Go to Window > Organizer. 2. Select "Devices" on the top, and "Provisioning Profiles" on the left. 3. At the bottom, select "Automatic Device Provisioning" at the bottom, and click "Refresh". If you've never done this before, you'll need to log in with your ADC username and password. 4. Give it a second, it should pull in your profiles. 5. Go to your project, select your app target and select "Build Settings." Scroll down to "Code Signing." You may need to go to "All" from "Basic" in the predicate selector. 6. Under Code Signing Identity, select the dev profile you just created. Note: don't use the wildcard one - it doesn't seem to work. Whew. Now, if everything went as planned (and you sacrificed a goat to Tim Cook and Tim found your sacrifice pleasing) you should be able to launch your app with no errors. But help! I got a weird failure on build! If you get a failure on build that looks like this: Command /usr/bin/codesign failed with exit code 1 Then it is possible that your developer certificate is set to “Always Trust” in Keychain. It needs to be set to “System defaults” for reasons that escape me entirely. Note, this may not be entirely accurate and may even be cargo-cultish. But I’ve encountered this “issue” twice now (once in December, and once now) so I decided to write down my steps so that, in a few months when this befuddles me again, I’ll know where to look for the answer.

Mac Oil Price Widget, Version 2.0 released

After a far longer wait than was intended, the Mac Oil Price Widget version 2.0 has been released. It was completely rewritten – like, I didn’t even look at the old code – and uses Bloomberg Energy as it’s information source. The display was also simplified – I really didn’t care about the chart in the old version, so the new version prominently displays the price and how much it’s changed. Download Here!

App Store Entitlements, and the Crippling of an App

A few months ago, I decided I wanted to try exploring the Mac App Store ecosystem as a developer. I’ve been writing little Objective-C apps for myself for awhile, and I decided I wanted to see what it was like from the other side. So I wrote this little app called Airplane Setting. It was a stupid simple little app that made it easy to turn off your radios with a single action. I wrote the app and paid my $99 admission fee. And after a month of back and fourth with Apple and a couple of rejections for what I consider to be dubious reasons as best (especially seeing as how I could point out existing apps in the store that broke the “rule” they said my app was breaking, but whatever, their store, their rules…), my little App was finally approved for sale. It did moderately well, passing 1,000 downloads with virtually no advertising from me. I had big dreams for this little app. Plugins, global hotkey support, localization, Applescript support, and more potential functionality. But all that was dashed by “Entitlements” and Apple’s requirement that all apps must be sandboxed. Look, in theory, the idea of sandboxing an app is not bad. The problem here is Apple’s all-or-nothing approach to sandboxing. The selection of entitlements are just so limited as to be nearly useless for anyone creating a unique, new or complex app - especially one that requires hardware access. Your choice is either to sandbox your app, choosing from the available selection of entitlements, or not sandbox it and not be in the Mac App Store at all starting in March. There’s no reason to only provide such a limited subset of functionality that a developer must choose from. Would it not be better to provide us a wider set of entitlements and allow us to justify our reasons for needing them when we submit our app? The reason Apple gives for requiring sandboxing is to prevent “rogue apps” from destabilizing the system. But when you consider that the App Store itself is curated, this requirement makes even less sense. If Apple is curating the store, how does a “rogue app” end up in the App Store? I’m a huge Apple fanboy. I have almost all Apple hardware in my house, from my iMac to my Macbook Pros, to my iPad and iPhone and my wife’s iPod Touch. I had AppleTVs before they were cool (and there’s one on every one of my TVs now). I love Apple. But as an developer … I [expletive] hate Apple for this “innovation” that crippled my once-promising little app. So, at this point, my options are: Leave Airplane Setting in the App Store. Doing so will mean no further updates so I’ll likely cease development. Remove Airplane Setting from the App Store and start distributing it exclusively from the website. My original intent with Airplane Setting was to explore what it was like to be an App Store developer. I guess … now I know what it’s like to be an App Store developer, and living in constant fear of Apple as a sword of damocles hanging over your head.

Goodbye, Eclipse.

Dear Eclipse, We’ve known each other a long time, haven’t we? I remember when we first met. It was way back in 2005, two jobs ago when I was working at interactive Point of View. I was still a young, naive kid, just out of college. At the time I was just getting my start writing serious PHP code, and you were a breath of fresh air compared to what I had been using before (Dreamweaver). You seduced me with your awesome power and functionality. I used to love being able to have code on top and a browser window underneath. Ironically enough, one of my favorite features would eventually be something I couldn’t care the slightest about. Later that year I would move on to Asteria, and I took you with me. This was the first time I had two monitors on my desk, and I kept Eclipse in one, and a browser in the other while programming. Again, your raw power made complex tasks easy. I discovered Subversion integration, which made Tortoise (I was still on Windows at the time) irrelevant to me. Your Subversion tools turned me into a huge fan. When I moved jobs again, to dealnews, I again took you with me. Much to the chagrin of my coworkers, I preached the gospel of Eclipse. When I first started I was still in the Windows environment and my setup was much like it was at Asteria. Later that year when I switched to Mac, I again took you with me. You occupied a place of honor in my dock. We upgraded together. Through Callisto, Europa, Ganymede, Galileo, Helios and Indigo. We upgraded through Leopard, Snow Leopard and Lion together. Sure, we had our occasional disagreements and outright fights. I remember one time when you would absolutely choke on the size of dealnews’ code tree. I would try other editors and IDEs. I tried jEdit, Coda and TextMate. But I always came back to you. But all things change, and this time I think we’re finally through together. The first sign you were no longer interested in me was the dropping of the official PHP build - the one I had been using for years. But you knew I was worried - you even said so on your website and pointed me to PDT - PHP Development Tools. This aphrodisiac, you told me, would make our relationship just like we were kids again. But what you didn’t tell me was that PDT would make you crazy and unstable in the worst kind of way. Your behavior has become increasingly erratic whenever you take PDT. You developed bugs, including ones that I could no longer justify. Ones that were literally costing me time every day. You said PDT could auto-complete code and when it does it works great. But when it doesn’t, the display glitches up the file so badly that the only way to get back into a usable state is to close the file and reopen. Now imagine doing this four or five times for every file you’re editing, every time you try to auto-complete some HTML. Your ill tempered behavior is costing me time and money. I tried to talk with you about it, but all you could say was NullPointerException. So, I’ve thought a lot about this. It’s been a good six year run, but I think it’s time we ended our relationship together. The truth is that I know about your other boyfriend, too. I know his name is Android, and I know you guys have been spending a lot of time together. And I’m okay with it. Really. All things change and we all have to adapt. The truth is I’ve been fooling around some with your cousin Netbeans, and I think we’re really hitting it off. In many ways, she reminds me of you. The difference is, Netbeans has herself together, is trying hard to improve herself and hasn’t forgotten who her friends are, instead of getting strung out on PDT and spending all her time hanging out in the backseat of Android’s Pinto. So goodbye, Eclipse. What we had was wonderful while it lasted and I’ll always treasure our time together and the memories we made. I hope your new life works out. Maybe we’ll see each other from time to time, but I honestly I don’t think that would be fair to Netbeans. She’s my new IDE now. -Rob Peck Eclipse User, 2005-2011