Perl "Out of memory!" when processing a large batch job -

a few others , happy maintainers of few legacy batch jobs written in perl. 30k lines of code, split across maybe 10-15 perl files.

we have lot of long-term fixes improving how batch process works, in short term, have keep lights on various other projects depend on output of these batch jobs.

at core of main part of these batch jobs hash loaded bunch of data collected various data files in bunch of directories. when these first written, fit nicely memory - no more 100mb or so. things of course grew on years, , hash grows box can handle (8gb), leaving nice message perl:

out of memory!

this is, of course, poor design batch job, , have clear (long-term) roadmap improve process.

i have 2 questions however:

• what kind of short-term options can at, short of throwing more memory @ machine? os settings can tweaked? perl runtime/compile flags can set?
• i'd understand why perl crashes "out of memory!" error, opposed using swap space available on machine.

for reference, running on sun sparc m3000 running solaris 10 8 cores, 8 gb ram, 10 gb swap space.

the reason throwing more memory @ machine not ideal solution because of hardware it's running on. buying more memory these sun boxes crazy expensive compared x86 world, , won't keeping these around longer year.

the long-term solution of course refactoring lot of codebase, , moving linux on x86.

there aren't generally-applicable methods of reducing program's memory footprint; takes familiar perl scan code , find relevant specific situation

you may find storing hash disk-based database helps, , more general way use tie::hash::dbd allow use database dbi supports, won't hashes values can references, such nested hashes. (as thissuitisblacknot has commented, dbm::deep overcomes obstacle.)

i presume perl code crashing @ startup? if have memory leak should simpler find cause. alternatively may obvious initial population of hash wasteful in storing data never used. if show part of code sure able assist