hadoop - Design questions for long running yarn applications -
i trying write yarn application , hoping suggestions on few design questions had in mind. have gone through simpler sample apps distributed shell , variations of familiar basic api. create application has web interface user can interact , potentially provide kind of tasks (nature of tasks irrelevant). based on work, ui requests containers processing.
the ideal arrangement comes mind application master provides web ui , no containers allocated until comes website , requests work. @ point, should able register new containers , allocate work them.
- if provides web ui , understanding chosen rm every time application submitted rm. means can have different ip and, therefore, different url upon application restart. behavior suggests should not used such purpose , potentially different application (non-yarn) can provide web ui , better suited it?
- in examples, have seen sample yarn apps requests containers part of invocation. can please point related apis allow requesting containers @ later time or potentially lets modifying resource requirements (memory) of claimed containers or increasing number of containers on demand
- similar last point, examples focus on yarn application , end. can imagine application make sense continue run forever (as web app). these long running application client rm api changes. ok disconnect client submission job process or start using & run in background
i appreciate suggestions
in regards question (1) can run in unmanaged mode. allow run outside of yarn cluster on dedicated machine ip address have more control over.