Generally, if you are up very early and can carry your own luggage off, departure will be fastest. Otherwise, times will vary. If you have an early flight you could get in one of the earlier departure groups, but in any case, departure is pretty efficient and you shouldn't be waiting 'hours' unless there is some kind of issue at the port, which happens infrequently but does happen.